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software quality assuranc2= and an in Vv 
selected industry quality assuranc2 functions and the Flise 
Material Support Office (F4SO) Quality Assurance Div 
Ouality control at FMSO is effected by the organizational 
element that produces the product and by a small, central 
iz2d stati. Improved systems develonoment and a higher level 
@eeqguality contrel are tha goals of FMSO. The recommenda- 
tions and conclusions offered are based onan extensive 
literature search of existing material on softwares q 
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The computer industry has gradually evolved over the 
years from the massive hardware systems that were very 
expensive to build and maintain to *+he present state of the 
art wher? a tiry one quartér inch square chip has more 
computing power, is enormously ch2aper to buy, and can be 
Maintained virtually 100 percent of the +ime. Along with 
the hardware change we hav2 seén an ven greatec improvement 
=n the scftware development from sinpl® mathematical compu- 
tation +o the launching and recovery of manned space 
venicies with greater reliability and performance than at 
any other time in history. 

This evolution has not happened by accident. The devel- 


opment has be¢n based upon making mistakes, documentin 


(4 


those mistakes, and procesiing on. AS people continued 


it 
reo 
(p 


process they studied past documentation and conzinued th 


(Dp 


@ecumenzaticn precess until it nas dDaccome an accepted part 


or the development cyclés. Although no systematic approach 

was utilized, there has been an 3ver inereasing tendency 

towards the development or a set Of standards that could 
h 


provide an avenue for common understanding with a mininum of 


confusion. 

The growth of systems software which occured during ths 
past two decades has presented 2ach organization with 
continuing challenges in maintaining an effective organiza- 
Seonal structure and ina Pel iow ince se fia cre me systems 
development methcds and strategies which can be transferred 
from one entity to another with a maximum degree of cohe- 
siveness and precision and a minimum cf ambiguity and 


eon fusion. 





NeEeGesaang to D. Rosse OMmesoaeacn 8nc. [Rete i, ene 
feels y Of Software is relative <co the S 

aa can Only be achieved by a distipli 

which quality requirements are in a 
original requirements definition for the problem an 
then carefully checked and confirmed at every stage in the 
production process. In order to hav2 any sensible treatment 
of quality every aspect of the system life cycle must be 
besed on an orderly, controlled, and disciplined metho- 
JdeVogy. This is not td Say “hat all software must be 
produced exactly with the same set of tools and technigues, 
for this clearly would be 2xcessive, Teese N2ayece 2 troad 
spectrum of the degree to which the ideal system technology 
is approached, tut even the sinplifiesd, streamlined me*ho- 
dologies must be complete and consistent. Each version must 
be, in some reasonable fashion, a proper degeneration of the 
Elaborate, most advanced state of the art, merely simplified 
Pomsui= @ Simpler set of circumst2nces. fe ust. 0S 2heun— 
bent upon all persons that are engaged in the project to 
combine company standards and common Sense to 2rrive 4* the 


Begqusred level thet will 


pe 


SUgS a Sue 2.) produc: 


Meeeccomplish this, there xists within each organiza- 


(D 


Seon discussed in thizt paper 2 group ahose rinary 
responsibility is ensuring that company standards are 
enforced. The Cuality Control/Y Quality Assurance Branches 
are the organiza ‘ions tasked with the job. be 1s thererores 


eee purpose of this paper to Examine the Quality Control 
programs of varicus government and aon-government organiza- 
tions that produce software and have established, well 
documented standards. The effectiveness of their program 
and the method of operation within their company structures 


will also be discussed. 
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momen 2 eae atenorcuws ll list and identify current 
feenas ana State Of She art processes, techniques, and 
methods that have been collated through an examination of 
current literature about quality and the software develono- 
ment process. In Chapter 3 and Chaoter 4 we will discuss 


the Quaiity Control programs at TRW, General Electric, Naval 
Fle 


Office. Finally, in Chapter 5, a set of recomendations with 


Oc2ans Systems Command, and the et Material Support 


fU@scs fication will be provided for FMSO consideration during 


the planning and execution of future corporates policy and 


expansion with emphasis on the Quality Control 2ffort. 
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The rapid expansion of the computer industry in the past 
ten years has been accompanied by an increase in the problem 
of producing a quality software product. Surveys have shown 
that as much as 60 percent of softwar2 produced have serious 
faults in the first ité¢ration - faults serious enough to 
cause the prograr to f21il in its task [Ref. 2]. A tradi- 


Meonal races of American manufacturing has beén strict 


emeeence tO qualziy Ss taniatas ani <he prediction of scit- 
ware should be no different. ALohough “COlmmMesr SGttwate is 
Sometimes consSicered more of an art than an eéngineered 


product, computer professionals a 

assurance iS an important part o 

and most data processing iepartments 

Seecuality assuzance function. “Many of the 
| su 


elements of production engineering 


testing, ovroject management and quality assurance are now 
faering aS an integral pert of a software production 
a? Such things as structurei prodaqramming, software 


engineering and quality assurance ars fast becoming the norm 
rather than the exception. User satisfaction, compliance 
mem approved methods of building applications, organiza- 
tional goals, and performance goals have all been driving 
forces behind this movement. Pisa, Gua pecer  OSeqs. With 
quality assurance in tha csomputer industry and its role in 


software development. 


WZ 





A. QUALITY AND CUALITY ASSURANCE 


Eohocey © land. =suD) 


(D 


Quality is, at best, jec 
measurement. The American Heritage Dictionar defines 
Mma cy 2S a@ characteristic or attrib “a PE®perty. Ibe 
also is thought cf as the natural or éssential character of 
something. In everyday life, quality measurements are done 
Soncanuaily and usually without second ROM. 
Side-by-side comparisons of objects under identical cconidi- 
tions and with predetermined concepts form the basis of most 
comparitive judgements. Gnfeortunately, these déecisiors are 
usually unique and have little valu2 to anyone else unless 
they are made by an expert (Ref. 3]. One widespread 
mememae, by 4ts very nature, quality defies definition and 
must be uniquely defined for th2 itan in question by 
alist cf characteristics and attributes. iy soe 
implies no evaluation or judaement of the item, but 
provides descriptive traits by which ‘the appraiser may form 
aamopinzon { Ref. 4]. Ken Johnson, 2 software quality ass 
Trance manager in industry and chairman of a working part: 
Set up py the Electronic fngineering Association concerned 
with software quality 2ssurance, disagrees somewhat with 
Sewer definitions concerning the ephemeral attribute of 
Gera ty and declares that quality is "the totality of 
features and characteristics of a product or service which 
bear on its ability to satisfy a given need. PeoeshOne, 3. = 
Mema ctaitrness for a purpose at an economic cost" [Ref. 2}. A 
recent study pointed to a number of myths concerning quality 
in organizations. These include such things as quality is 
impossibie to measure, quality lowers productivity, quality 
means poor workers, and quality is th2 responsibility of the 
"quality department". It goes on to give the sharpest defi- 


feeton of quality-"quali« is the sum costs of prevention, 
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ill not as comprehensive as 
mew be» desired. Hence, Se eeeecec cis CONCeErnenS guality 


measurements must be tempered with raalistic knowledge that 
any measurement will be partially imperfect or imprecise. 
When dealing with softwares, in itseli not the most tangible 
of products, confidence levels and error tolerances play an 


important role in determining acceptable quality levels. 


2. Quality Assurance Defined 


—= a ee =p <P eoe oe oP 


Oiesiiwe the concept Of Gualety, Wiecae ss JSuially 


meougnh- ©f aS am attribute of a good or service, gjuaiity 


t~- 


assurance is most often related to 2 process or nethodolcgy. 


Mesora=ng to Frank Ingressia from IRW Corporation's DJerens= 
and Space Systems Grcecup (DSSG), "quality assurance is a lot 
like sex, freedcm and democracy. EVOEYORe 25 fOr 22,3 Mout 


only under certain conditions." Theres are a host of defini- 
Seems Which apply «to quality assurance. Mie. “Of Greta l, his 
Force position is that Juality assurance is 
whick provides adegquat® assurance that mate 

supplies and services conform *0 established technical 
requirements and achieve satisfactory results [Ref. 6]. 
This definition is much nore encompassing than most early 
interpretations which callad for guality assurance to merely 
verify conftcrmance to speci fications. At the other end of 
the spectrum is the feeling that quality assurance is merely 
mae business of ensuring that the product is not the result 
Of good juck but rather the inevitable creward for good 
Management practices. Still anether definition and perhaps 
on? that will pecome widely used has been proposed by the 
Institue of Electrical and Electronic Engineers in their 
recent study concerning software quality assurance standards 
(Ref. 7]. Tt states that "quality assurance is a planned 


and systematic pattern of all actions necessary to provide 
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meegiace COnt2dence that the a2teSm or product cornfetms to 
established technical reyuirements". tees, Con net.oOn has 


4 


Beene DOL cOWed fzEcm MIL-STD~-1098 and is ccrsistent with <h 


( 


rm 


accepted usage of the tern. Coniortang to specifications, 
ensuring good management practices, testing of requirements, 
confidence that the system is reliable or the product is 
desirable--all of these things ar? considered goals of a 
quality assurance plan. The bottom line for a quality assu- 
rance function is ensuring that user's needs have been 
ad=quately satisfied. There are a multitude of different 
philosophies concerning the achievenent of these gaoals and 
eh-y wili be briefly addrassed in a Later section of this 


chapter. 


Meeeitbadttional Otiality Assurance in Industry 


For manufactured products, quality usually means a 


combination of quality of design and of manufacturs 


{Ref. 8]. Quality of design is the value inherent in the 
design; a measure of the axcellence of the design in rela- 
mon to the custcmer's requirements. The gquelity con=rciler 
has the responsibility ¢5 ensure that the quality level 


determined by management can be achieved on ovroduction 


equipment. Quality of manufacture is a measure of how well 
mies product, at acceptance, conforms to the design. There 
are most often five basic stages of quality control ina 


maccOry { Ref. 8]. These sonsist of: (1) Deciding what to 
Manufacture and frepare sp2cifications covering all réequire- 
ments, (2) Make pre-producticn checks and work out 
organizational responsibiiities, (3) PEOQuuce: ol, (4) 
Feedback on guality deficiencies, and (5) Establish long- 
term quality plans. 

Bhemmatalsey Conzrol Eunction in a manufactuting 
Pet 2S usually a very complex and intricate organization. 


It becomes involved in all phases of the production process 
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The role, structure and 2b jectives *qualaty assurance ia 
Peaewale PpEeOaduc. 10m Will bs @€xamined in detail later in the 
chapter, but there are many Similarities which exist and 
indeed industrial quality assurances forms the basis for 
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Pigure 2.1 Typical Quality Assurance Organizational Structure. 


software quality assurance technigues. Pigure 2.1 iilus- 


trates a typical manufacturing guality assurance department. 
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Be. THE ROLE OF QUALITY ASS URANCE 


1. Why the Need for Juality Assurance? 


WW 
1 


The statistics concerning the growth of the software 
industry in the past 15 years as well as the problems 
concerning this growth are quite well documented and recog- 
nized. For example, one company, Bozing Aerospace, repcrted 
that ona large software project: a). only 14 percent of the 
total number of runs would have been required had there been 
no errors or failures, ani b). 39 percent of the runs, while 
successfully compieted, were later invalidated because of 
Mae =rLOrsS, tape failures, Or program bugs {[ Ref. 9}. From 
a cost standpoint, over 2ight billion dollars was spent for 
software of various types in 1980 [Ref. 10}. Compucers, and 
in turn software, are becoming entangled in every aspect of 
our daily lives. Peels) eCctronl.eG Danking and Shopping <o 
NASA Space projects to more effecient use of farm machinery, 


we rely on computer software to help us make more and nore 


ry 


£f our decisions. Software developers have recognized *znsi 
fa 


te 


Tesponsibiiity towards gqualit software and mcs 


yw) 


a 
processing departments now incorporats some sort of quality 
meerance function ints the production of software. Taas 
quality assurance function is able t9 alleviate the probiem 
most software managers have of becoming involved in the 
system at a point when the cost becomes significant and the 
dates of implementation approach. The establishment of 2 
quality assurance function provides managemen< with a deaqres 
of confidence that an independent, technically trained group 
is monitoring the goals, methods and performance of applica- 


Pons trom =he beginning of the project. 
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2. Objectives oF Quality Assuraace 
The quality assurance function, as part cf the 


Systems production group, works *5 ensure that standards 
concerning goals, methods and objactives are met. The 
quality assurance group *ypically performs those functions 
+hat the data precessing nanager might do personaily if time 
permitted. Quality assurance reviews each system to ensure 
compliance wiht the following items. The system must mest 
Meemnecds Oc the user d2partment ani Otner users and at the 
same time not infringe on the rights of other systems users. 
The goals of the system should be consistent with the objec- 
tives orf the entire organization. Bow krere LS a COns!2ct, 
the goals of the organization should maintain priority over 
the goals of one user. The systsm goals should also mesh 


With the EDP department objectives and if there is 


ity pv 


conflict, it shcuild be resolved before implementation. I 


ai 
iD 


there are external industry or government requirements, + 


Meals Or the system should conform to these standards. 


Gen-lOls on the system must 53 complete (management 
controls) and the system must be auditable. The system 
Sould conrtozrm to all geénearal policies, procedures, stan- 


dards and guidelines established by the organization and «he 
electroric data processing department. Quality assurance 
mus= finally enstre that the design 3f the systam is econon- 
ical (least cost system), effective (desired results with 
minimum effort), and efficient (maximize use of people and 


machines). 


The cost of a quality assurance function is very 
difficult to estimate or 2ven measures. William Perry, an 
author of extensive material on software quality assurance 


and a member of the Quality Assuranc2® Institute in Orlando, 





Meets da, comes Closest tO a precis= figure by saying that 
a 2 


"if quaiity assurance is included as ibelee: lzem in 32 
Poo jec=e's budget, it should range somewhere hetween 2.5 and 
BePeErCcent Of the total project cost" { Ref. 4]. BeGure m2 ez 
is indicative of the percentage of costs expended on a 
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Pagunce 2.2 Quality Assuranc2 Project Costs. 


typical five-phase scftware déveiopment project. This esti- 
Mate is still rather idealistic because of the differences 
which may aris2 because of staffing alternatives, netho- 
Gemogy, iitfecycle entry, the difficulty in defining quality 
assurance and other unknowns. 

S@scen dus titicat2jon iS an importer t aspect of imple- 
menting a quality assurance functior. In the hardware 
arena, the cost cf quality assurance is most often justified 
by lower warranty cost markups in tke price of the product. 
This is equally true in the software world. Warranty costs 
Will be lower for a gqualitv softw#ar2 product. Another 
fringe benefit of a quality software product is it+s ease of 


-= hd 


adaptability to a similar product at 2 lower cost (Ref. 11]. 
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mee COS< Savings or an efrfic 
is cften hard to justify becaus 
those not quality-consciosus of? 
improvements very difficult. AS is usually case, 


t 
results speak the loudest. The cost savings involved in 


+S 


having projects done on time and within budge allows 


quality assurance tc maintain its level of efficiency and 


reduces management time spent sorting out the mess which 

results from bad planning. 
Finally, it has b2zen shown th 

the quality assurance function in ir 
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Meoject has 2 


< 
EGe Comeeer= =~ COST 
Beope and structure of the quality as Nee effort iis 
meerected strongly by the cost of errors related to the ohase 
Sede veicpment. Figure 2.3 is a typical illustration of the 
economics of error d¢étection in the various a of devel- 
opnent. As this figure shows, the earlier a roblem is 


detected, the less expensive is the rost of correction. 


SE SOFTWARE DEVELOPMENT 
1. Software Lizecyci2 Phases 


Referring to the lLlifecycls of software is the most 
common method of addressing the development of a softwares 
product. A review of the lituraturs has produced a plethora 
of illustrations of what is considered to be the true or 
idsal lifecycle. Most 2xamples use iifferent terminology to 
describe what is happening at a particular stage in the 
ihmeeccycle, but they all tend to includ] the critical items. 
The simplest lifecycle found is one in which there are cnly 
three stages - design and deveiopment, active and passive 
freer. 13]. Conversely, Bip MOSeacoOND Ve meade rinitwen of 4 
software lifecycle contains eight phases-systems definition, 


software allocation, Specamaca clon. design, code, 
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moure 2.4.3 Relative Error Correction Costs. 


foe ices On, integration, and operation (Ref. 14]. Tables 1 
depicts a comparison of the various phases of the lifecycle 
of software. 

Whatever the phases of software a¢cvelopment ars 
called, there are certain items which must be accomplished. 
The beginning of a project may be designated as a4 ‘feasi- 
bility study, requirements definition, systems definition, 
user requirements study, initiation study or something else, 


Maeeit consists of all activities which deal with deter- 


Mining whether or not a software project shouid be 
mies ated. Such things as cost-benefit studies, goal defi- 
ers Ons, and documentation requirements are «ypical 
activities which should b2 accomplished here. Next comes 
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TABLE I 


Scftware Lifecycle Conuparisor 
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AUTHOR ie sCYCLE PHASES 


PERRY ------- Feasibility - Design - Programming - Testing - 
Conversion 


aoe. -—--=——= SOTAN Design - Requirements De 
esign - Code ane Checkout - T 
Integration - Operational 


WEN DLS =------ Design - Coi= and Debug - Qualification Test 


raoekTS ==-—-— Design and Development - Active Stage - 
Passive Stage 


Requirements - System Functional Specs - 
Sceewear> Puncirs onal Specs = : | 
Tmpiemantation - Varification and Test - 
Operations, and Maintenance - 
Configuration Management* 


PIPS PUB 38 - Initiation - Devalopment - Operation 
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Sign, design and development, softwara design, or 
u 


oe 
a 
4-4 
ee 
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ie 
systems functional specifications. Aci as 
design alternatives, specific ree fInMet Tons oO. 0e 
performed, and program and data base specifications should 
be included in this phase. The next phase is probably «the 
moSt rudimentary in terms Of work ani déals with programming 
and testing. Thess sStage tS elso Esierred *o as coding and 
debugging, verification and validation (after coding is 
complete), Or is sometimes broken into two distinct phases. 
The system is now written in the desired language and 


Various tests ar¢ performed to ensur2> the system performs as 


desired. The next, and most often Last phase, is referred 
to as conversion, integration, operations, implementation, 
maintenance, or configuration managament. This stage 
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UB, perfom@ing ongoin 
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Bemsitsts Of Maint2ining the softy 
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4 
Beyabdatzons and Changing lt aS additional requiremenzis 


2. software Properties 


As stated previously, softwar2 is an elusive product 
upon which to plece a quality measuriment. Howley and Fink, 
software quality engineers for the Boeing Aerospace? 
Ser poration, have attempted to verbalize what a quality 
software product is by stating "A quality software product 
May be defined as one which exhibits the following proper- 
tiss: 2t satisfies the software spe 
requirements; it performs all in teen 
melmetavely free of design, interface and coding d¢fi 
cies; Poe nasmedueslow lz: fe-cycls sost Ss 
identified and documented; and it incorporate 
eee ewabre Quality chamacteristics" [Ref. 15]. Tf 

vel of guality which is desired by the abcv 
there are anumber of* factors 
Memec™ = Quality Software FRef. 16]. These include, 
ier are not limited to: 

1. Correctness - This generally ausans programs verforn 
in exactly the manner specifisd in the pregram docu- 
mentation. Correctness is usually considered an 
ideal quality which is rarely achievable. 

2. Reliability - This attribut2 means that proarams 
De=rOrm relatively trouble free all the functions 
expected from the specifications or documentation. 

3. Validity - Validity is concerned with the question of 
whether the functions and performance of the programs 
are adequate and suitable +9 a needed purpose. The 
software, without manual intervention or additional 
programming, should perforn Shemerumece tons  <Jhas 


reasonabiy would be expected of it. This attribute 
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2s avery subjective one 2nd must be flexibis <9 
changing requirements. 
Resil: S sould be 


ja. 
(D 
> 


res aw be oe 


feo Seca ncesna > DEC gran 
designed in such a way to be forgiving of commcn user 
and data errors. Inconsistent or unacceptable data 
entries shouldn't provcke actions which make no senss 
xo the user. 

Usability - Human factors and limitations and conve- 
hnient usage techniques should be considered whenever 
@ program is written. 

Clarity - Programs shouid b2 easily understandable 
Irom the users manuai and all 

should be clear, concise and cogent. Programs should 
be modularly designed, hav> e¢xpvlanatory ccmments 
where necessary, and use meaningful choices of vari- 
aple names. 


Main*aina bility - 390d documentation and comments 4 


7 


well as clear structure will make programs mor 


(D 


O 
easily repairable. Ghanem sea USO neSSentival for 
NaAKLing minor improvements. 

lied@enedutiaty = Major changes should be anticipeted 
and the softwares designed so that program functions 
that might require major change are well documented 
and isolated in distinct modules. 

Generality - Programs shouli be applicable +o a wide 
range of input valu2s and usage nodes. 

Portability - Programs should be easily adaptable ¢t9 
Sransrer to anoth2r computer system or operating 
system. 

Testability - Programs should be simply structured 
and use gsneraly algorithms, to facilitate step-by- 
step testing of all capabilities. 

gificiency + The attempt should be made te keep the 


cost of program oparation as low as possible. 
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Bae preceding isemmeontains many ®oG “ene attributes 
of a quality software product but is not the only lis* avai- 
lable. Hehere aie nors, thecHude Sich EhingS aS integrity, 


mexabilaty, reusability, interoperability, and others which 
are descriptors cf quality softwares. Figure 2.4 is a good 
illustration of how these Factors affect each other and whaz 


degree of a certain factor is required when a different 


BectOr 7S recognized. AS can be seen, some factors are 
synergistic while others = Onmalans The impact Or 
Pewclict2ng tactors is that the cost tc implement will 
increase. This will serve to lower benefit to cost ratios 
{Ref. 17]. 

3. Hardware Characteristics is SORB wens 


ee SP aes See Se 


Characteristics 


Hardware is a tangible piece of engineering. It has 
Very precise specifications and drawings and is based on 
Peemerostabliished building principles, the aim being to manu- 
Bieeicte Many identical (9r near identical) items. The 
design-development-preduction cycle is mazure ard 
tuned. Refinements t9 the design may be made many «ime 
before a commitment to manufacture is made. i GOntEase, 
software engineers ship their prototypes. SOttware 25 a 
largely intangible product, only described by many volumes 
Of Specifications and listings. Software is unlikely to go 
through as many prototype stages and, therefore, the cppor 
tunity for design iteration and iaprovement is somewhat 
limited [Ref. 18}. 

Most aspects of hardware ar= functionally testabl=2 
and nave very specific requirements tasting programs. It is 
fenced-in by established principles and well-known, widely- 


used disciplines. Unlike hardware, software is functionally 


Ut 


non- testable in all but the simplest of computer programs 


and as aresult, Sees “Very dirrrcuic tO test sottware 
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Figure 2.4 Relationships Between Software Quality Factors. 


completely [Ref. 19}. Figure 2.5 illustrates “he problem of 
testing software. Each ciccle represents a processing nede. 
The clockwise arcs are jumps around the individual nodes, 
and the counter-~-clockwise ares specify the number of itera- 
tions of each half. The number of discrete states possible 
Within this trivial diagram is approximately 100 quadrillion. 

If these could be tested at the staggering rate of ons 


per microsecond, it wouli have been necessary to start ths 
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Figure 2.5 Computational Paths. 
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Meenong ovet 2,000 ysars 2370 to neset next month's 
delivery. 

The usual cause of hardware failure is component 
deterioration. Software failures are almost always design 
errors that show up only when th2 software is used under 
certain conditions. Hencs, Quality Assurance techniques for 
BEemeware focus on getting the design right [Ref. 20]. 

A comparison of hardware and software lifecycles is 
offered py Tabie 2 and Shows stiearly that Similaz terms in 
the two fields scmetimes nave radically different meanings. 


4 


The driving forces behind imvleméenting management 
Seeucture in any organization are reduction of costs, 
m@emeased cOontrcl, and production of a quality product. 
Pemcwale pEoduct ion is no different. Figure 2.6 d¢picts the 
rise in softwar¢ costs in the last ‘twenty years. As 1s 
apparent, software costs greatly excaead equipment costs over 
tha useful life of computer servic:s. Reece S Kapa ot 
growth it is imperative that softwares be managed to minimize 
Goasts. While cost minimizaticn is an important aspect of 
software management, there are other reasons of sequal impor- 


tance. Most software in production today is @ compiex 
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TABLE it 


Hardware and Software Product Life Cycles 
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Develop Manufacturing 
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WanwidGelic re FOCuct 
Make Product Available 
to Users 


Maintenance (Correct 
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Correct Desian Flaws 


Enhance Product 


SOFTWARE 


Determine user 
Requirements 

Develop Product. 
Concept (Functional) 

Specify Component 


Design (Detailed) 
Implement and Test 
Programs 
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Available to Users 
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cations is such that programming 
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activity that must be directed effectively. The 
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Figure 2.6 Estimated Growth sf Software Costs. 


Meeouncability of project decisions and objectives and gives 
top management visible measures of success in the accom- 
piishment of goals. As 2 general rulé, software projects 
Ss often initiated by management personnel who control 
budgets and scheduies and software designers frequently end 


ndependen= work wit! 


UpeedOlirg a substantial anount of 
littie pressure to evaluat? their progress or the remaining 
Meek OL COSTS. With a software managment structure iin 
Beece, important issues and objectives such as cost, quality 
and schedule can be carefully evaluated and the appropriat= 
jmeewonsibiiity for the decisions assigned. BeGuinwea: 
con*=rols, working procesiures and resource management are 
further justifications of a strong software management 
Srmeeucture. Questions about systen feasibility, system 
quality, design methodology and testing procedures are ones 
which should be answered by software management. Ali of 
these things help designers and programmers to organize and 


Gieec. their efforts efficiently in solving such problems 


De, 





meen avVallabdle COS: and time jiimnits. Another reason for 


software management is th epgoung probleweros MA: ieenencs 


(p 


Q@azter inplementation. Some estinates put the cod 
maintenance at 70% of total software costs. It is in 
+9 recognize that maintenance is just as important as 
opment. Good software management principles will atte 
be ongoing throughout the lifecycle and will make the 
strongest effort to stress close management of effort toward 
the most needed software capabiliti 


The management of software development is often 


rererred to as "software engineering". Th2S Luplees chat 
the principles cf production engineering managemen* n D2 


ie 


ca 
ransrferred or applied ts software development. 12ap 


Gn 


engineering suggests 7} 
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Softwa 
"the entire development of a soft- 
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Ware product fates anetelal (CONnceo ton papolpaio, UICS/ Jel 
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Should, tneref 


ct 


e 
asystematic and manageabl2= fashion. TI 
Meee oss Die =O MCRStOr ths" quaicty, pestormance and cost of 
*he end product ‘tnroughk the several phases of its lif 


Sxeie'” fRef. 21]. 


5. Szandards for Software Develjoment 


An area which has been serisusly neglected in the 
software development industry during is growth has been the 
establishment of standards cf conformance. There has been a 
Meesogni==Oh of this lack during the past few years and 
attempts have beén made to provide alaquate standards. fas 
moSt widely used military document concerning standardiza- 
MIL-S-52779A dated 1 


cable to Departnent of 


1) 


meen Of gualit assurance plans i 


August 1979. This document is app 


vt x. 


Defense agencies when acquiring software where the acquisi- 
tion involves either software alcene or software as a portion 
of a system or subsysten. It provides  atiick guidance 


concerning software quality assurance program requirements 
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ana cOVGers Such things as tools, techniques and methodci 


gies, ccmputer program design, work ce 
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Q 
Onaat 
eS) at 
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oe On, computer pregram li 
audits, configuration management a —- testing (Ref. 22}. 
tieescdociMmen: 2S useq not only by DOD, but has also 
been referred tc by many civilian organizations in the 
absence of anything better. The Institute of Electrical and 
Electronic Engineers has recently sponsored a commiztee to 
evaluate «he problem and develop a s2t of sofiware quality 
assurance standards. Thair stated purpose was to "provide 
uniform, minimum acceptabl?2 regquirenesnts for the preparation 


and content of software Jgquality assurance pians" (Ref. 23}. 


@ 


The sections of the standard developed contains direction 
Concerning such *hings as refsrenc2 documents, management, 
documentation, standards, Prace ces, and “Conven <1oORne, 
reviews and audits, configuratiosa management, problem 
MmeLtling ard corrective action, tools, techniques, and 
Memencdaologi<es, code control, media control, and supplier 
gan =Ol. Nome Mw =- 2 ete "o> = Ncl Vo mands comprenons.v> 
memecOope and 2xrovides guidance for jlzvelopment of a thorough 


software quality assurance plan. 

The preceding two documents 3aze the most widely used 
standards referred to when developing a software qualit: 
assurance plan. There are some other publications which can 
be reviewed for direction concerning development of sofit- 
ware. Federal Information Processing Standards Publication 
38 (FIPS PUB 38) provides thorough 2nd comprehensive guide- 
lines for documentation of computer program and automated 
data systems. National Bureau of tandards Special 
Publication 500-11 is a good overall guideline to computer 
software management and guality control. There are a host 
of journal and proceedings articles dealing with software 
Sati ty assurance which provida information. All of theses 


are not "official" guidelines and lack any authoritative 


Se 





endorsement. The best hops for better results from softwares 


Poe ty e@sSsSsurance DrcgqtamS ites in 22 accertance sf tn ISEL 


Standards and conformance sage their reqgquiremen=s. 
Unfortunately, these are relatively new and there is insuf- 
ficient data to declare that new industry standards have 


been developed. 


D. SOFTWARE QUALITY ASSURANCE METHODOLOGY 


1. Quality Assurance Planning 


O 
i ex — _ 7 : Ml - —s aa = -_~ as - S ; . e — i 
SeMEwe eS OUSCS arc NUSt semain cynamic to be of ary use. I* 


m 
is important that plans are modifié: 
y 0 


requirements as the eeu. ality 
assurance plan must indicate the particular activities which 
will erable the required ilavel or quality to ba achia2ved on 
any giver project. How the product is to be assured and 
teste acc. W¥ities the quality assurance group is to undertake 


in order to satisfy organizational requirements are key 
2lemen=s. 

Quality assurance generally ovaraileis the systems 
development process. The position of review points 
depend upon management's requiremenzs concerning decision 
points or information requirements. The importance of ths 
system to the organization as a whole will determine the 
amount of time sfent on 2:ach project. me mGr aierea DOs hes 
are the end of ¢ach systems development life cycle phase. 
At this time, an opinion is render2ad by qualizty assurance as 
to the adequacy of the design process up ~t9 that point. 
Mies OPiGi0n can then be used as a decision making factor in 
Getermining whether to progress to the next phase. ie) 
discussing the EeVvView ~ Cr teria a five phase systems 


aifecycle will be assumed. 
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The feasibility study commonly consists of 2valua- 

*ion of alternatives and techniques <0 solve a particular 

problem and reccemmend a course of action *9 management. 

This may or may not include a computer system ard he 

personnel involved in the study may or may not have itiieioes 
2a 


experience. The role of guality assurance during tne si 


+o 
(D 


Beet y study is usually one of a consultant +59 discuss ¢ 
practicality of alternatives or cost estimates. At the end 
of the feasibility study, quality assurance should evaluate 
whether the study team followed th organization's proce- 
dures in developing a proposal for management and comment on 
any computerization aspects of the —— 

The next phase, design, is critical to quality assu- 


UaeR this phase and 


= 


rance. The qreatest impact is nade 
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the quaiity assurance group should Srv Poet mpacr , fa> 


Yi 


design without actually participating in the design process. 


The goal is to not argue for or against particular designs 


but to review the proposed design on neric. One mexhod 
which is widely used in this phase is to ivide the design 
SmeO =WwO oOnases-informal and formal. She. n= 


ormal phase 
Seeurs after a oreliminary design i¢ on paper anda consists 
Seeaqual:ty assurance giving discussion only review +o allow 
S@eme deSign group to de<armine if they are on the right 
track. There is no report to management generated from this 
phase. A structure such as this requires a good working 
relaticnship between quality assurance and the design group. 
A+ the end of «he design phase, a fotmal review occurs and 
the design is normally fixed at this point. COlptLanceras 
performance criteria, Systems goais and oprocedur2s are 


reviewed by quality assutance. 


ct 


Program design and program coding make up the nex: 


phase which must be evaluated by quality assurancs. While 
these two activities may b2 combined into one phase, it is 
usually more effective and Meer aci i lateates ~structured 
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programming to separate then. 37 re 
design, ‘quality assurance has a O 
ensure compliance to design procedures and standards. This 
will hopefully alleviate many problems usually encountered 
in the coding phase. At the end of the programming odnase, 
quality assurance should perform a review to ensure compli- 
ance to procedures and standards for such things as coding 
and use of operating systanm facilities. This should be a 
detailed review and quality assurance must examine all 
aspects of coding, operating system instructions, fae 
structures and anything else which will affect the operation 
of the computer. 

In the next phase, system testing, quality a a 
#5 mostiy concerned that an adequate test pian has been 
M@eepared, <hat it is followed and that it conforms to the 
standards of the organization. Quality assurance will only 
review test results to ensure compliance to standards and 
should not become involvei in a detailed test pian. Az the 
end of the system testing quaiity assurance should again 
review for conformance t9 organizational policy and check 
for user satisfaction. 

The last phase in our example, implementation, can 
be tne broadest in scop2 and iongest in duration. Gis 
phase is sometimes called conversicn and is general 
thought of as the process of replacenent cr new installment. 
AS with testing, the primary concern of quality assurance is 
that a bonafide plan has been defined and that it is being 
followed. The completion of the implementation phase brings 
the final review by quality assurance that *he procedures 
defined in the design stage were followed. Once again, user 
Satisfaction is of paramount importance and quaiity assu- 


rance is reviewing plans and procedures. 
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The foregoing example of a quality as 


S 
over the lifecycle of a software product is by na 


means the 
cnly one to be used. Another typicai example is Giscussed 
by Marilyn Fujii, a software quality assurance professional 


mem LOg2con, Inc., indwhich the lifecycle is divided into 
seven parts and quality assurance again has a role over ths 
entire lifecycle [Ref. 24]. The early stages 2f the plan 
are centered arcund defining the procedures and standards 


which wiil be applicable to supporting configuration manage- 


ment and computer program development. Most other 
S@ecavizties consist of reviewing and auditing sotware 
products against previously set standards. Qualizy assu-~ 


rance is responsible for all design reviews and audits and 
they evaluate all documentation such as test vian, specifi- 
cations, and users manuals. Any walkthroughs or 2ccepntances 


testing will be scheduled and conducted by quality assu- 


rance. Meet new adslsVery DOLNE In he Lifecycle, they audit 
jase cine. contiguratiion c> be instailed in the operational 
eryironment. Pigure 2.7 offers a visual presentetion of 


quality assurarcs's role in this example. 

The examples given are but two of a multitude which 
can be found in profession2l iiterature. Regardless of what 
Specific method is used, there are 2 number sf components 
which should be included in any software quality assurance 
pean. The plan shculd identify procedures +> be used in 
issuing work tasking instructions for all work relating to 
software development. Monitoring of vrocedures and assuring 
adherence cis. then should be part @) ie any plan. 
Identification of schedules and resources and tracking 
progress toward them shouid be included. Work descriptions, 
responsibility assignments, initiation procedures, report 
generation procedures, and scheduled completion dates should 
be addressed. The plan should document quality assurance 


involvement in the specified development vrogram. Also 
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Figure 2.7 Quality Assurance Activities. 


provided shouid be visible schedules, milestones and 
interdepartmental dependencies and commictments. Levels of 
detail required should be included in any plan. Finally, 


tha organization responsible for software quality assurance 
Should prepare the plan. A genéral outline of a typical 


quality assurance plan is provided as Figure 2.8, 


Beeotatting and Organi Zati on 


The success of any quality assurance function begins 
Ween the personnel assigned to the staff. Individuals as 
knowledgable as senior systems analysts and designers should 
idealiy be assigned +c quality assurance. I+ is not ¢nough, 


however, for quality assurance personnel to merely possess 
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Figure 2.8 Typical Software Quality Plan Outline. 


the characteristics of 2 good systems analyst. Qualict: 


na 
assurance personnel must command the respect of both the 


individuals whose systems ar2 being evaluated and the 
management of the EDP department, who must rely on then. 
The ability to review ths work of others and to convince 


a 
them there are fetter methods to perform their work takes 
some unique skills in dealing with people. Quality assu- 
rance reviewers must hav? a talent for good communicative 


and persuasive ability as as well as be respected for their 


Meennical ability. 


a 





There are a number of variables concerning the 
experienc? and numbder of psopie astigned to quality assu- 
Tance, EASE SIiZe CE thewemetect et hand is A major 
determining factor. Smali projects which appear to be rela- 
tively simple and perhaps repetitive of previous jobs may be 
short-changed in the quality assurance area. If the company 
itself is small, it may not be able to afford the commitment 
to quality assurance of a large corporation. T9p managemen* 
may not recognize the nead for quality assurance and hence 
give it less than prominent attention. 

The organization of a quality assurance department 

e+ up in many ways. CONGR tery Deldera=oay ois tha 
the higher in the data process SUSueEtuce- she. Gual=t 
Meeaence Luncticn reports, the better the probability of 
success. Also, the level of reporting is sometimes indica- 
tive of management support [Ref. 4], Figure 2.9 shows quite 
a simplified view of a rapresentativ> EDP departmen* with 
Meemauala ty assurance function placsd as a staff function 
reporting to the EDP Manager. hits —Staue ture 2ansurss She= 


ue. y assurance will craceive the acttenticn fies DP 
Manager and 


that it will be independent of all other aspects 
of the dat pro 


cessing department. Figure 2.10 shows ar 
@egenizacional structure from industry= Informatics fInec. 
Quality Assurance in this organization is embedded in the 
organizational structur= and has secondary aifects 
throughout the company. As another example, Feqguce 2.14 
shows a varlation of the placement sf the quality assurancs 
mumietion [Ref. 25]. This structures, with quality assurance 
e@emen Independent function reporting directly to a division 
general manager, provides good ind2pendent oversight. 
Different projects within moc Gani gewoon faweel 1 
receive different emphasis in the yuality assurance area. 
The same hoids true for industry as 2 whole. Enbedded soft- 
a we 


ware in weapon system will not raceéive the same sort of 
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quality assurance atctention as a payroll prceject. AS ¢ach 
Beoqvec. nas differing requirements, SO Well “each cquaizcy 
assurance scenario be different. PieEOudroulr, ears cass 
concerning quality assurance implementation, there are 3 
Humber cr methods used to orgenize a quality assurance fiunc- 
e200. These have been consolidated inte four general 
methods and will be described in som2 detail. It must oe 
renembered, noweéver, that the type of organization of tne 
@mai2=y assurance function will depend gréat 
factors as the type cf product and industry, e?mphasis 
quality assurance within the organization and the scope of 
mene project. Quality assurance departments from selected 
companies will ke examined in detail further in the pa 
Meme n Wiil expand cn the methods of organizing the quality 
assurance functicn. 

The first method, and probably most widely used when 
Organizations are beginning the quality assurance function, 
=s the task force? method. Uitcm= cada! lows DEgan:Za =: ons 
to become involved in some sort of quality assurance 
Memevety prior to the formalization of the quality assurance 


function. A task force offers the advantage of developing 2 


Shy, 





_ — er eee ee ee eee = 





| Products Management 


| Technatouy | 





| | Software Seen 
| 
i 




























Support 
Services 


Technical 
Surocrt 


Product 
Support 


Product 
laintenance 


Product 
| Development 


$ 






a He 
: Pee | i | 
a en ; [| [Seren eee) 


| ! 
a Education Z Headguarter ! | | Data Processin 
Cs ies Encineers | Services 


ee | | 





— 
—<—— 


www wm Oe — lm 





SE gg err i carr carte SE cect SON eteins aS ete ee en On oe 


- 


= oe ——ee 





EE TS ie ee ee ee eee 


Figure 2.10 Entopmatics Enc. 8rganization Chart. 


group eable to handle the? unique problems which may 0»be 
encountered in a given software project. Task force members 
with the appropriate background can be hand-picked by the 
EDP manager. Another benefit is the training afforded the 
systems designers asSigned to the t23am because it puts them 
in a position of analyzing the competency of systems design. 
@ea2sadvantage tc this method is tha problem of continuity. 
Each task force will tend to develop its own methods and 
procedures for the review. Tf the task force members are 
Meee reiseved Of a Significant amount of the burden of their- 
Gaily work, then ancther possible problem is that they may 


have trouble finding adequate time to devote to the review. 
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A second method is the formation of a full time 
quaizty assurance staff. This method prevides the greates* 
amount cf continuity among reviewers. The EDP manager can 
Meme heve a greater degree of confidence in the quality 
Sesurance function. By asSilgning a full time start, manage- 
ment is giving a Signal of the measure of importance it 
places on quality assurance. The biggest disadvantage of 3 
full time staff is the sompetency wi The ce vl ew ) GEOUD. 


Whereas &@ task force can add specialized knowledge, a full 
time staff operates with the personnel assigned. Another 
problem is the technical jolrete lie che slew ae keyed ene peta mes 
Meennicel proficiency with current practice is very impor- 
meee DOth from the standpoint of cradi Sethe Stare 
Meee the rest of the organization and the proficiency with 
which the personrel can perform their function. 

The permanent committee method is another approach 
and is basically just a step up from the task force method. 


Continuity of individuals is the biggest difference between 
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the two methods. Where 3 
committee will be convened 
Peo }ECtS. This says to project 
will be 


indicates 


reviewed, 


@ higher degree of 


specially convened task force, 


permanent committee has 


reviewers can devote to 


maintaining «heir workload. 
eed 


a+ 


is just a committee 


tiveness of a function staffed with full time perso 


Bole eneeneT hog 1S 4 


The 
Ber= time personnel. 
the 


Someinut=ay of 


use of one or more rae Sh 


ene Oo Wa legny 
Dyepart time personnel «+o 
advantade would be the ability 
as needed to review projects. 


mmarvidial should be named «to 
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au b ough © uk 
JNTO aAamsaubse-— 
previously stated, reviews 
of each development phase. 
review points 


inftiuence systems design as 


These reviews occur at the following points: 





1. Midjustification phase, 

oe Ena Of FUStitication phase, 

3. Business system solution phas2 
4. Computer equinoment selection, 

5. Computer system design, 

6. Program design, 

7. Testing and conversion planning, 
8. Program ceding and testing, 

9. Detailed test plan, 

10. Test results, 

11. Detail conversion planning ani programs, and 
12. Conversion results. 


Naturally, the number of review points would depend 


Om a number of variables including size of system, impact on 
mre Organization, and makeup of the quality assurance 
organization. ipeeddasson =o thess review points, quality 
assurance can perform valuable consultation while conducting 


the reviews. 

Mos+ authors are not as specific as Perry as to *he 
siming end placement of review points. The general 
consensus is that reviews must be oredefin 
points in the development process, b2 u 
thorough, and are conducted in accord = 
Peocedures. 

There are a number of tachnig 
assurance review team may employ durin 
review. When gathering information 3b 
reviewed, such «hings as project documentation, system docu- 
mentation, interviews, observations, ands tne sce of 
established checklists are appropriate methods of gathering 
information. Practices used when attempting to vaiidate «he 
information given during the gathering phase inculds 
testing, evaluating test lata, fornulating bases case data, 


meen nd Vidual confirmation. Macsc 8 ene InkOrNa=.0orn. 2s 
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Gatnered and validated quaiity assurance mu Evaiua~e tana 
data for management. This @valuavtison is typically besed on 
intuitive and evaluative judgement, mathematical simulation 
or modeling, expert advics, or quantitative analysis. This 
list is not exhaustive, but is representative of the types 
of techniques used by quality assurance reviewers in 


achieving fair and comprehensive raviews. 


Auditing is sometimes differantiated from reviews. 
Audits are usually thought to be final acts where ail loose 
ends in a quality program are tisd up. Tyoes, Of audics 


include in-house audits where the audit verifies that 
developer is adhering «£95 ali davalooment stand 
procedures, subcontractor audits to énsure that the subcon- 
me2ctor is complying with all software standards and 
Beaceduz=es imposed by the sontracz, and fect-finding audits 
me which the subcontractor is evaluated to ensure ke is 
Capable of furnishing reliable, quality softwares of the type 
desamed necessary to meet contractural requirements 


Bekins acer Or milicetg Onis and! Slectzical Engineers 


rs 
Qa 
a 


standards propose a certain minimum number of reviews whi 
Should pe conducted during the software deéevalonoment life- 
eyvele [Ref. 7]. These i ~ware reduirements 
review *c ensure 


n a r 
the adequacy of tht requirements stated in 
the software reéequiremen C 
e 


ations, a preliminary 
design review to evaluate the ta al adequacy of the 
preliminary design of the software, anda critical design 
review +o determine the acceptability of the datailed soft- 
ware designs. Recommended audits consist of a functional 
audit which is held prior +o software delivery to verify 
compliance to all requirenents specifications, a physical 


audit to verify that tha softwares and documentation are 


internally consistent and ceady for delivery, and in-process 
audits to verify consistency of the jiesign. 
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4. Testing 


Vouiricd smemeis an Cher word for testing. T+ is 
essentiaily ensuring that the conditions are as s 

involves doing whatever is necessary +9 verify that the 
statements or conditions are correct. Although correctness 
is the overall goal for most testing efforts, it is not 
always the overriding concern. Larg2 programs are sometimes 
so complex that they never completely satisfy their specifi- 
cations. Thesé programs may be quite usable because 


failures are encountered infrequently in practice, and when 


Mees OCCUr, <‘t2eir impact on 2 user is small. To be 
usable, correctness i5 not always necessary and sometimes 
mee goed snough. A corr2c® program may satisfy a narrowly 


drawn specification and yet not b2 suitable for operational 
use because, a meee ce, PapuckS Net. Satistyin the 
specification are presented to th2 program and cresuits of 
such incorrect usage are unacceptable. Thus, if a program 


its 


(n 
Ze) 
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Q 
f+ 
ry 
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Q 
ct 
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O 
7 | 
= 


2S correct with regard +o an inadequate a 
Meseeeress =S)Of Pttelie value. The oroblem which atises is 
m@pae MOS. testing consists of correttnres 


meeeitcctl= cesting done for reliability, rebustness, e¢ffi- 


ct 
x 
ey 
| 
a 


ciency, and other properties which nake a valuable sof 
product. Whatever property is péing tested, the tests which 


are valuable are *hose where tne result is not predictabie 


= 


so that application of the test and acquisition of «he 
Seeriececons -.tutes an intdrmation gain or a reduction in 
Maectrtainty [Ref. 26 J. MOmachiev> LaqeS goal, “tests should 
emeck the program at th2 boundaries of its behavior. [In 
order for software to be tested in the most effecient 
Manner, a test plan with complete procedures and methodolo- 
gi2s must be formuiated. Other than from the obvious reason 
of ensuring test efficiency, the reasons for this are to 


provide an audit trail of ‘testing so that future problems 
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Mey be ag2ssected Ecrrom the point thsy initially surfaced ana 
shat bpoundary areas wners testing w a 
May he easier to identify. 

Software quality assurance should become invelved in 
testing in a number of areas as illustrated in Figure 2.12 
Before the testing begins, it should ensure that ail soft- 
ware, hardware, and the @nvironment are in a satisfactory 
state and that test and simulation software have been 
defined and are under control. It should witness loading 
and running of the softwar2 and ensir2 the test resuits 
retained and all discrepancies noted. Finally, quality 
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M@m@acccms OU Wd waSo ese nl ne =£Uunci.5=s < 
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+ fl 


ions concerning deviations and disc 
Mebeo-o 272A, th> scandards £0 

used by the Department of Defense, contain 

ieee CL software «testing procedures f Ref. 22}. i 


BO. 1loOwLEc: 
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Ss 
2 2 
testing procedures are utilized by many civilian softwars 
development ordanizations and consist of the ( 

m 


analysis of software requizremen= <> deter 


(dD) BeveCW GE £6St ESqGuUlcSementS ant ctiteria for adequacy, 


’ 
6 


@emecbrlity, and traceability and satisfaction of require- 
ments, (Cc) review of test pl:ias, procedures, and 
Miele c2cacions for compliance with contractor and contrac- 
tual requirements and to insure hat all authorized and enly 
authorized changes are implemented, Ci “Vere Tr cation that 
tests are conducted in accordance with approved test plans 
and procedures, (e) Gob==-1Ga- On chat tes. Lresuivts ate she 
memema. findings cf the tests, (ff) caview and certification 
of test reports, (g) ansuring that test related media and 
documentation are maintained +o allow repeatability of 


fests. 
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Figure 2.12 Typical Functions of a QA Dept. During Testing. 


BS. Software Quality Assurance Tools and Technidgues 


Improving software development and <Eest processes 


@ 
Meenas in iarqge part upon the application co proper tooisc 
and techniques to the development Lifecycle. The aifferen- 
tiation between tools and techniques is very clear. A 
technique may be defined as a procedure for implementing 2 
Meliability or quality aqoal. Techniques consist of s<an- 
dards and procedures used in development and maintenance of 
software systems [ Ref. 25}. Suc enaNgds as © Structured 
programming, top-down d2sign, system modularity, proper 
language selecticn, abstraction, MNeOrMateon Aiding, and 
program design Languages are generally thought to be techni- 
ques in achieving softwar2 quality. 

mOOlS. cn the och=r hand, have been defined as an 
automated technique [Ref. 25]. Computer programs which 
perform measurement tasks which would otherwise have <+o be 


done manually are considsred tools. There area large 
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number cf tcois *o ensure software guality assurance on ths 
marcketplace. The probl2m is that most automated testing 
tools are expensive to instaii and use, -fiersme : tests 


require very different tools and most tools are incompatible 
wet each other [Ref. 27]. Sel2=canemeewot SpecLezerr tools 
should be done only after careful analysis of the objectives 
desired of the tools, the tools cost- funding and the criti- 
cality of the software functions t> be tested. Another 
consideration shculd be the phase of davelopment which the 


software is in. Figure 2.13 shows some typical tools which 
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Figure 2.12 Tools Used by Software Phase. 
may be used during the lifecycle of software development. 


While not a specific list of tools which may be utilized, 


the figure gives an idea of the types of tools found in 
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iniustry today. ebe t= fee xoteamat2on cf sele 
the above list follows. System modell 

whereby a Simulation of the hardware/software sy 
programmed uSing 4 Simulator. Interactions béetw 

ware, scfitware, and personnel are simulated and incompa 
System requirements often become evident after system 
modeling. PSL/PSA (Problen Statement Languag2?/Problem 
Statement Analyzer) -+This is a specific tool licensed by the 
University of Michigan, Project ISDOS. Tt provides a means 
for describing information, computer and software systems. 
A requirements data base puilt from several contributers can 
be checked for consistency and formal completeness. pvesign 
modeiing--Critical algorithms are coi2d in a representative 
manner to determine of <~he design will result in the desired 
accuracy and 2xécution times. femend.s Menory Marg. ns; 
resource utilizaticn, ani traffic rates are nodeled to 
ensure adequacy. heGuem=sien = S @facsability tso0l--Software 
requirements are linked +o successive design data base 
entries, test pianning, and test iata entries to provide 

Vv 


requirements traceability. Interactive debug toois--The 


4 


@eong =~O0l controls the code hile it is a2axecuted and 
2 


W 
displays memory and regist2rs. Th gisters and memory can 
be displayed while the code is Secured §1ns cr uc: On, bby 
instruction. Preset memory locations and registers hold any 
desired value thus allowing branches to be executed and <=he 
HoJ1C debugged. ES (interpretive computer Ssimula- 
mon)—--This tool allows =he instructions, interrupts and 
input/foutput capabilities to be mad= visible by simulating 
the architecture and memory of a larger computer. The 
program can be started and stoppsi in order to evaluate 
performance of the program at various points. Stress 
tests--As the name impli2s, this tool tests the computer 
under worst casé conditions of various parameters such as 
Nn, 


memory input rates, Memory Wea Zecl° S2Ce 
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Hardware/software test peds-<A test bed is the system i 


(he 
<, 
ib 
' 


hardware joined with the system software and combined with 
the appropriate test drivers, moniters and environmenta 
simulators to provide as near an operational system as is 
possibie. Regression testing--Reagression testing uses 32 
standard proven test for testing the software after é@ change 
has been incorporated in the softwares in order <9 detect any 
Side effects or errors due to the change. These tools ars 
but a few of the literally hundreds on «he marker. Tools 
can be a valuable and useful additioa +9 any software devel- 
opment life cycle but must be chosen carefully, checked out 
before commitment, and used in the proper perspective. This 
means that the tool should be recognized as a tool and the 
results should be evaluated carefully and action only taken 


On specific results generated by the tool. 
6. softwa 


Perhaps the weakest link in nodern software develoo- 
ment has been documentation. There are number oz sources 


a 
Seer formation which provide gquide concerning software 


iD 


1 S 
eeecumentation. Meee -S-oi27e 9h 2nd cth= LESuwquality assurane 
Standards both provide requirements for documentation. 
fee o-22//9R calis for all doc 
act 


programing conventions ani opr 


ntation standards and 
ic2s to be used for ail 
software to be referenced in the yquality assurance plan. 
eee tGBEE Standard calls for identification of the documenta- 
tion gceverning the dévsliopment and verification of the 
software and an explanation of how the documents ares to be 
checked. mePkuseher calls tor a number of specific docu- 
ments. These include a software requirements specification 
(SRS), software design description (SDD), and software veri- 
Meat ion plan (SVP). FIPS PUB 38 provides extensive 
guidance concerning documentation of computer programs and 


ADP systems. Software documentation is an extremely 
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mae Ge ese ec. Of SOftwWwa: es deévelopuent in thaz i= is 


a} 


‘D 


& 
}-! 
Ud 


means cf communication which the j2Siqner usés with 
colleagues, management and the technical authorities of the 
customer. Dipeeugnm it is) Widely £2scoqnized that good docu- 
mentation practices should be maintained for all system 
projects and there are ample guidlines with which to 
proceed, documentation remains largely inadequate. Much of 
mets Old, it is poorly written or written in such a manner 
as to be incomprehensible to the average reader, it may not 
be thorough or leave out 2lements which are critical to the 
Software in question, or it hasn't been changed te reflect 
Suecren= practices. Software quality assurance must realize 
the requirement for good documentation and take steps t09 
mere =hat documentation which accompanies developed soit- 


Meee 25 complete, clear, accurate ani concise. 


Configuration of the software systen at discret DOl nes an 
nee Mee Ol peSse 15° tO  Systemacically monitor changes <0 
Mees COnftigutation and maintain the integrity and trace 
emis cty Of this configur2tion <hroughout “the system life 
cycle. Pen es Mri Martly SOnG=rsed  Wlth = ensuring the 
moe egrit Siimecteinwety Of désiqn. Quality assurance 

mesough configuration Nanagement, shouid enrorce the 
eel ilowing: (jeeeeCOmctgubat2 On Tdentitication=-A system of 


Meceording the technical issctiption of individual computer 
programs and supporting documentation, thus documenting the 
functional and Enyce Lwcnaree ==risctcsS O: =he conficqured 
Mmeem., (b) Configuration Control-applies to configured soft- 
ware and documentation after they have been released. pre 
also provides a control for changes and ilibrary features. 
(ey Configuraticn tatus Accounting-the recording of the 


Status of the svstem's configuration. Pie tWwaepoOse 2s =O 
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A. INTRODUCTION 


As stated inf[Ref. 22] the 
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otherwise provided under the 
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of all actions nécessary vide adequat 
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feet. 29). 


abe e 
Cc abeBad 


Bom eley 
Sree norie: defi nwtion is, 


AMGCOGECeE TION OF G2tlcisenci2es and 
Overall quelity performance" [{Ref. 28]. 
of these 
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they all have 
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Sheught of easily, 
@i>-omet complaints. 
Quali+ 
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y assurance achieves thi 


his 
This con function is based on the exi 
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Smep.san and the control function i 
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Baee tO that plan. Effective control 


from the plan early, before 
control detects deviation as it happens 
in effective control are 


(2) 


she plan can be 


Two key factors 
of the 


which progress on 


plan and astablishment 


these milestones, action can be ini 


tial deviation from the plan. 
In essence, 
be added later 


not the job 


quality assurance is not 


of a single perso 


is added at 


or greup 


that quality +he cen 


Se. 


purpose of a 


sofiwarea 


G@eLtneslOns car 


Same goal - 


Goal 


stence of 


when 
ae 
of milestones 


measured. 


in the software development 


time and in <hse 


quality assu- 
devaloped, 
GOrn tract COmMpL.es 


AtOenes sere n = On 


Lea noughepses and 
be 
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EHEOtgh ~Gojecen. 
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Simply ensuring adher- 
will dets 
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Si aeVva eo. 2On 
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total knowledgs 
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amount. Oiled SsclvenseumbDegins at «he Start orf <=ne 
Cevelopment process and iS Continualiy added at ¢ach step 
along the way. The primary job of the quality assurances 
group then becomes the development of the quality assurance 
plan and once it is developed, the management of it 


throughout the development process. 

The next sections of this chapter will discuss standards 
which are in use for developing quality assurance plans. 
Following that will be 2 discussion of the tvpical softwares 
development cycle. The final section of the chapter will 


Giscuss the quality assurance programs in use at two major 


7) 
©) 
(i 
ry 
qi 
(I) 
t 

ti) 


Civiiian companies and at the Wavai Jce¢an Sys*en 


Be SOFTWARE QUALITY ASSURANCE PLAN STANDARDS 


S2779A 


{73 


ieeetetitary Specificat top 


Mil-S-52779A applies to the acquisition of software 


Peenet alene OF aS part of a complete systen. Le. Geauzces 


th 
ray) 
in 
Oo 8g 
bh 


the establishment nd implLementacion o are quality 
Sesurance program by the sontractor. 

PoimegicDemoece Of Mal=5=S2779R deals witch the sort- 
Mane Quality assurance plan. According to this specification 
[Ref. 22] any scftware quality assurance vlan wili include 


the following areas 


A BOs Techniques and Methodslegies: What are =hey 
and how will they support mew Overall. Qudlicy 
Assurance Program? Examples include: | Operations 
research - Svstems Analysis, Funcyiona and perior- 


mance requirements analysis, “=r Or Saaveis. software 
Optimization tcols, specification tracing and coding 
conventions. 


7a Ccmputer Program Dies gnaw ent “design ) logue, 
fulfillment of requireménts Some leteness and compli- 
ance with specifiéd standards b e evaluazed? 

Se wWOrk Certificat cSOn: HOwaewit?) the description, author= 
ization and COMPS <2 On Of work be certzfied orf 
approved? 

4. Documentation: |. What. documentation Standards and 
program cenvenrntiorns will be used? 
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PeeomPM=->  ETOGrdam La otany Controls: How will ditigrant 
COmpUtEr oSEeR Wee oweie c= 5) oGePtitled anc  s0cu- 
mented? the ecstive her® is to insure that enly 
aporoved SRW Ce S Gee are mai2 and implemented. 

6. Reviews and Audits: How will reviews and audits. be 
conducted Oe eoSsu re Peace] apt t2 2 y rom sl ipyst a bre We 


requirements to final product? 


7. Configuration Management (CM) 
Quality Assurance and CM relate 


? 

e following areas - 
a) Analysis of requirements to 
b) Review of test requirements 


adequacy, feasability, trac 
tion of requirements. 


ieesiecria tO LASUES 


d 
8. Testing: This section inciudes ¢t 
d 
a 
zability and satisfac- 


h 
etermine testability 
n 
a 


c) Review Ror test Platoon ec lures mend Speca rca — 
ens -O PCeMl a sce Win Son tractual requizersnts 
and to insure 23ll authorized changes are imple- 
mented. 

Deveras.cGetien of tests. 

Neeccrtipicat ton, Of test results. 

£) Review and certification of test reports. 

g) Bee eee of test material to insure repeat- 
a0L Tt ye 

h) Support software and hardware used during develop- 


ment must be accevtable to the go vernmen?. 


2. i 


lizJ 


E 


Tes 


Secnaakr 


oes 


Jie 


tI- 


ty Assurance Plans 


6 


ofswar 


Ta?) 


223 


ft dy 
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Meecwoulmpese Of this standatd is to provide uniforn, 
Minimum acceptable requirements for preparation and content 
of Software Quality Assurance Plans. The standard applies 
to)6 the «development and maintenans2= of critical software 
ae where failure could impact safety or cause large 
Einancial or social loss2s). For non-critical software or 
software already developed a subset of the requirements may 
be used [Ref. 23]. 
The following are the najor sections and subsections 
of the pian as outiined in [Ref. 23]. 
1. Purpose. 
2. Reference Documents. 
3. Management. 


a) OTganizaticn 





b) Tasks 
c) Responsibilities 
GQ. Decumentation 

Se ues Po Ss e 

b) Minimum Required Documentation: Software 
Requirements Specification, Software Design 
Description, Software Verification Plan. 

CG) Gener s Comput2cr Program Development Plan 
Configuration Management Plan, Standards an 
Procedures Manual. 

5. Standards, Practices and Conventions. 


a) Purpose 


b) Content Docum? nt ie aar ee HoOgsc _S et euGceuce 
Standards PeSocete 7 cee cas, Connen sasy Standards. 
6. Reviews and Audits. 
e) Purpose 
b) Ms nimum Rquiremants: Software, Requirements Review, 
reliminary Design Review, Sritical pes oe Revieaw, 
BeAiGe tHe dT INEUGLIe Physical Audit, in-Precess 


Audits. 
7. Configuration Management. 
cee PFE cblem Reporting and Corrective Action. 
9. Tools, Techniques and Methodologies. 
ieececae COnNe rol. 
fie Media Control. 
i. Suppiier Control. 

If any of the above sections are not pertinent to 
he project for which the plan is being written, a@ statement 
stating this non-applicability should be included under «he 
section heading along with reasons why it is nor applicable. 
i= additional sections are needed they may be inciuded at 
the end. 


C. PHASES OF THE SOFTWARE DEVELOPHENT PROCESS 


It is generally agre2i upon that the software develov- 
ment process consists of at least th2 follewing sever phases 


(Ref. 24] : Conceptual, Rediiremeanes Netinzet2on,  Desiqn,; 
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ecomemcqmend Cheemelc, IeSs:lad,eemecszyzation, and Operaticna.. 
Software definition takes oilace in the Conceptuai phase. 
This consists of feasability studies, trade-off studies and 
analyses to define specific reyuirements to be allocated to 
computer resources. Once these regquiremen*s are defined and 
documented they form the basis for a iraft systam specifica- 


tion which will te used during the following phases. 


During the second phase, Requiranents Definition, it is 
determined which system requirements will be implemented by 
software. Through analysis it is determined which software 
functions are needed and the inputs, processing, and outouts 
mre abe required for each function. Also par* of this phase 
is <t=he Finalization of the system specification and the 
pr2paration ‘ene the drafz software requirements 
specification. 

Following the Requirements Defini 
phase of the development cycle, Design. The object of this 


phase is to come up with a software design *hat wiil imple- 


ment the functions identified during the Requirements 
Meeanicion phase. The design will include actual algorithms 
pede equations alcng with control logic an date sperations to 


be performed. The finalizaticn of the software requitements 
Meecerication and the preparation of the draft so*tware 


design specification will also take place during this odhase. 


Qu 


The fourth phase, Coding and Theckout, includes trans- 
facing the software design into a computer orogramming 
language. Usually itis a high-order language but it may 
alsc be assembly languags. OpeeseoNpsta=2en Sard essenbly 
errors are corrected each individual progran module is 
executed to remcve obvious errors. This procedure consti- 
tutes the checkout. 

Once coding is complete the fifth phase of the cycle 
Testing, begins. Here the software which has been developed 
is tested to shew that it is consistent with system and 


sottware require ments 


oe | 





Dipaag eins egrcaticn, hardwazrs end software are brought 
Ss 


together and system and operational testing is conducted. 
meemcpjece Of the testing is to insure the satisfaction of 
system requirements in th2 actual or Simulated environment. 
The last phase of the cycle is the Operational vhase. 
The software has been accepted for use and the only 
remaining activities ar2 Mainten2nce and modification. 


Figure 3.1 [Ref. 24] shows the software development process 
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Figure 3.1 Software Development Process. 

and the key outputs of the phases. Pagipe  Sa2 | Ret. 28 ] 
Saews how “he quality assurance activities fit into «he 


development process. Figure 3.3 [Ref. 29], although it does 
not specify all seven phases decrib2d above, it dees show 


the activities of not only the Quality Assurance group but 
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Figure 3.2 Software Quality Assurance Process. 


Pemene Groups in the projact srganization. The @ctlVeu2es 


listed are all quality 2ssurance related. 


D. QUALITY ASSURANCE PROGRAMS 


1. TRW Defense Systems Group 


@. Background 


Kurt F. Fischer (Ref. 30] writes: 


At TRW Defense and Space Systems, gqroup the need for a 
division wide dganization toa assist software management 
became quite apparent quran Cho Sebastes 1960s. Like nost 
other software vendors, RW was concerned about th 
frequent cost overruns and schedul2 slippages in it 
software projects, and decided t9 develop methods ¢t 
Suen, onas trend around. AS part s£ that decision 2 
established its first quality assurance staff in 1969. 


a), 
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At TRW the software quality assurance functions 

Meee perireoz=med DY an Crganization tltied Pr 
This organization is head2d by a vi 
a 


directer. Figure 3.4 [Ref. 30] 
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Figure 3.4 TRW Corporate Organizational Structure. 


organizational structure. The Product Assurance ozrganiza- 
tion acrtually performs a dual role: Quality Assurance and 
Configuration Management. The reason for ‘this is that both 
areas hav¢ been found to share similir characteristics. 


1. They perfcrm staff »ariented functions. 


Ze Ine perftermance of their functions is often times 
more credibie when done by 2n independent organiza- 
1c ep nee 

3. Stafrt personnel share many aiptitudinal characteris- 
mecca mond eec More athentionec> detail, preference for 
wide visibilty tasks) [Ref. 30]. 
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Puemmect  ASSurance on {he produce level is keaded 
Pyean ASSistanhe Program Manager ror Prcduct assurence whos 
has responsibility fom Dern Quality Assurance and 


a0 
Configuration Management. The APM for Product Assurance 
receives his direction from the project manager, yet he and 
his staff remain organizationally independent from the 
Beegect by reporting functionally to the Product Assurances 
Seganization at the corporate level. Figure 3.5 [Reft. 30] 


meustrates the froject organizational structure. 


9 


De. Quality Assuranc2 Objectives 


- oe pa ads 
Lom aer te yor sf. S- 


objectives, the following activities 
By the Quality Assurance group at IR: 


1. Assure, direct traceability between the subsystem 
Sipecetecet ons and the develcooment Speci itications. In 
addition assure direct tracsabilicy between develop- 
jem Spee 2h2Cac2Ons and =ne =2S- plan end trem thers 
to tes procedures. OAGwee sade tionally insure that 
all requirements ar2 traceabl> «0 the product speci- 


meee = 1 Ole 


2. QA will develop and maintain a software requirements 
matrix. This matrix will be maintained throughout “he 
scftware development cycle. 


Seni. dOCUMenta {ion generated iuring the project will 
be reviewed by OA personnel. 


4. Conduct audits of the software development process. 


5- QA _persennel will participat= in all formal reviews 
and audic (.@.9. Software Requirement S Review, 
oe Be Ty Desigjn Review end Critical Design 

evilew 


6. Insure the implementation of built in checks and 
balances. 


fee Dersonnel will monitor and witness the Preliminary 


Oualification Test (POT) and Formal JWadeetcas on 
Test (FQT). These cest result ee be cosigned by QA 
personnel. All QA test records ill be maintained. 
Sie A perscnnel will MONItOCL +he eo2 = gure ton 
anagement peGurec-cmmaur ng  SOttware development. 
They fo eso =st tO Insure the integrity of <h2 
software configuration. 
Bee OS Personnes will participate .in all. Configuration 


Change Board (CCB) meetings and ovrovide a revi ew of 
all propo sed changes . in .the software development and 
test process tO again insure the integrity of <+*he 
software configuration. 
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Bagure 3.5 TRW Project Organizational Structure. 


TO . 


11. 


1Ze 


13. 


CA weil. BUD BOs tha development of project Software 
Standards and Procedures. 

Verify that all requirements and functional cépabili- 
ties have been satisfied by software testing. 

QA personnel will insure that the delivered software 
package meets contractual requirements. 

A system for tracking Software Problem Revorts (SPR) 
and Design Problem R@ports (DPR) throughout the soft- 
ware development and Peete 1On spo! Wis be 
developed and maintained by QA personnel. 





ee eme tala one NeGCessery inplementation and suppor: 

aoa OA anc Ce eters es, Qualixv NSSusensc= wre 

ROS OG ee Mime scheeclon —OEOceSS Of all scfttware 
COOLS. 


c. Quality Assurance Planning 


The successful implemantation of an effective 
Quality Assurance Program relies heavily on quality assu- 
rance planning during the early phases of software 
development. At TRW QA planning is accomplished by a4 
complete review of the early project documentation. Examples 


Sf such documentation include the Contract Statement of 


wD 


emer System SbSc2f2cacions end PFs ject Plan ancng others. 
Once this initial review is complated the Quality Assuranc 
Pian iS prepared. This plan contains «he functions, tasks, 
and tesponsibilities of tha Quality Assurance group and @i59 
identifies the quality assurance tools needed tc insures 
PertWa=e quality in the areas of accountability, teste 
memes ty, wUsabidaity, maintainability and relia 

completion the plan is reviewed by other project organiza- 
tions and approved by both the Program Man 

customer. Upon approval task assignn2nts are 


out the activities outlined in the previous su 


- | 
3s OQ Oo 

qt 

rw 

O O 


noted in [Ref. 30] these assignments are based 09 


effort and must remain flexible to adaot <o: 


1. The needs of the surrent ph23se in the develcpment 
Pants — Cie ea. 
Pees Chet eS IN attention needing areas (.eg. teGhnieas 
problems). 
3. Unexpected demands placed on Quality Assurance by «he 
Project Manager. 
Once the QA Plan ZoeeopEOveds =he QOvualzety 


+ 


Assurance polocies and procedures ara written describing tha 
methods and procedures t%) be used in the implementaticn of 
quality assurance requirements defined in 

fae «6S yScem Specification. 


Zs Contract Statement of Work. 
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Bee (Pe O4 coun dan. 


GD egnO he fa TEESE 
Gd. Software Standards 


Bowne eho raise met Potnts Out in (Ref. 30], 

"The purpose of software standards is +o improve the main- 
tainability and readability of the software product". TRW 
has developed a comprehensive and detailed program to deal 
With software standards. According *> (Ref. 30] this proaran 


hes been successful for two reasons: 


ie SOL cware Standards are not. dictated from the execu- 
eee ) OL EAGSS, OF from Quality Assurance, but are 
Neve peagwscU= Of Cicse So womeliGe=eaon anon g 2s 
Design, Development, Laome on amd SrOqect. Offcces. 

2. A tool has been provided Ona d-ONdmmeatlly. checks = he 
SCeewere agea=rst QOS < Gz  =hs Sceneards. This allows 
programmers to audit themselves so that there are no 
Surprises at turnover time. 


The Software Standards and Procedures (S.5P) 
feet. 31] decument contains the gquali*y provisions, instruc- 
fMeenS end Standerds fer 2ach project. The SSP deals with 
Standards concerning software, firmware, design, development 
pea testin The categories of standards included are: 

1. Seurce code formatting standards 


Z. HOC  aoSlbes +o be used in software/firmware design, 
code, test and update. 


Se tandards dealing with QA +9091 development and their 
use during design iavelopment and checkout. 


Waivers and deviations serve to complement «he 
Standards. They allow permanent or temporary relief from 
compliance with the standards due to technical difficulties, 
inefficiencies or schedule impact. All waivers or deviations 
must be approved by both the QA Manager and the Project 
Manager. 
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2. Quality Assurance Tools 


Mie wise Or SOLEWare tTOOls Should only be consid- 
ered where it will prove =o be more cost effective and nore 
accurate to have the task automated rather ‘than performing 
it manuaily. Tasks which may fail inte this category are 
often tedious, enmacdka ios ings,. difficult, Error prone, 
repetitive and cestly [Ref. 30]. 

Alt NGion “RW tS Cubtently using improved and 
updated tools the following are examples, taken from 


fmret. 30], Of the types of tools in use at TRW: 


fee P2oduct Assurance Confidence Evaiuator (PACE) = PACS 
ees 8G co =o Glen al sacs faaly Sa t= 54 Slee OU Y 


and rigorously a program has peen tested. 


2. FORTRAN Cede Auditor (FCA) = This tool audits coding 
Standards and by doing so allows enforcement cf thesé 
Same standards. 

se Sesue ured Srogqramning Auditor (STRUCT) Saysyibice Uh Gabe sis 
used to _ensure progtams comply wit the structured 

rogramming standard. It 25 executed after PACE 
€eausSe 2+ belies on Gutpuct Erom PACE. It evaluates 
Peegecam structures based on the following Six 
COR Smale cas sequance, if-then-else, de-whila, 
do-until, case, |escape~-rrom-lsop. 

4. Units Consistency Analvsis (UCA),. - This is used on 
POmiesniesouree Cods and the assoOcia-~esd data base.  i* 
Scans the code 205, slieirS se pkeetes Setlaeee NS « By refer- 
SG ea temcocd Dase fou <h> versebles used in sans 
eg@uac  ORNw i= Geccr Nines =f she UNL=S in ghe assignment 
statement are in fact consistant. 


f. Quality Assurance Reviews and Audits 


Review and audit points are established during 
the QA planning phase to ensure that design, code, inspec- 
tion, testing, and documentation are compatibles 

As uséd at TRW teviews serve 2S quality assurance critiques 
or documents while audits are ritiques of processes 
Peeeuat.on critetia for doSuments [Ref. 30] consists of the 
Beellowing: 

1. Adherence to format and pagination. 
we Clarizy of objectiv2s. 


me Teehni calscontent. 
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re rniterdocument consistency. 
5S. Traceability to higher level specifications. 
The audits conducted bv Ole tys SASsule ce 
personnei [Ref. 30] serve the following four functions: 


fe Reel assess ccmpliance of source code ard documenta 
=on to software standards and procedurss. 


2. They assure traceability of requirements. 


3. They determine th2 satisfaction of system require- 
mEnts during system test and acceptance phase. 


uW., They assess test sufficiency. 
phe following List contains the audits conducted 
by the quaiity assurance personnel at TRW. A brief descrino- 


mon OL each audit is also include:. 


ie, Unt Development, Folder (UDF) Audi - The UDF is a 
non-deliverable item which orovidss a mecharism for 
Zieeckwal cometrol 2nd also e vides managemser~ Visi- 
Dawie=y mer ~Sottware Ee opment. The UDFs= are 
prepared and maintaine ol each softwares ,UR Lt 
provided by the oi Coee Mre Jet2nition Of @ unit is 
a mg SC SOUurZOe OF A Wrou Otetogtea tt related 


routinés. The UDF isa coli Coe ONO r. ad Soqdlae 
merts, design data, code, and test data pertaining to 
Gwe rec" ftc UNLt. The UDF serves as the primary 
SWevoullance —Mecnanh Sh PtOr =ne duality assurance 


Demscnnel Guring. =ne d2sign, Sedeng and Wines tease 

Dhases of the software devsliopment project. tach UDF 

Semeud wed. Phas 20dt2 3s divided into thrse phases 

ae p se orc sO pra yide early datection and corzec- 

Leen OmepeSss = bie SsYrors. f#32ch phase is designed t9 

elie. a SPeCcific area oF the development process. 

POmloOnoNde2s 4a G2Scription (Ref. -31] of sack phase: 

a) Phase fI- veriiies appropriate UDFs have been 
gece 2 LCC, eee o cover sheets and inserts haves 
been inciude requirements have been stated in 
the requirements section and that the design 
Secu Ol CoOmrtatns tre CUETeNt WOrEKing design. 

b) Phase II - A desk check and SOR eG. eens audi 
is performed to determine if code isbei J produces’ 
in acccrdance with established project standards 
and guidelines. 

Cc) Phase, III - Verifies aero acn sD PD ald ced 
contains a Coo eesOne Of. teSc resuits. ,and 
analyses necessary to demonstrate that the unit of 
code has been debuaged, anit development. eee 
is comrlete and that an up-to-cata design cdescrip- 


tion exists. 


2. Test, Data Folder. (TDF) - fhe TDF i 
working dccument for the test group. A 
provides a surveillance vehicl? for qua 
personnel. DieeAg Seene  antexracion a 
testing phases of the development pro}: 
contains the test requirements, tesi 
procedures, test exécution reports 


the primary 
such it also 
ity assurance? 
qd acceptance 
Gi. The TDF 
ia) Esha +est 
and Software 
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ee cbiem reports. Quality Assurance reviews the fciier 
insure adequacy SemMmecconssS and CONnlOrn2 cS. TO 

standards outlinéd in the Software Stanéards and 

Procedures (SSP). 

Cle gre Sie jblie el agalrey al See ete Audi - Quality Assurance 


Usesecitoesalart £6  monLtor the GORELGUEAcTION Menage- 
Henegiceimmemres, CO 1NSUre tha: they éomply Ween DOtH 
wie wc Danan end joe@nmanzad proacedures. 


Interface Verification Audit - This audit, is 
conducted as early as possibls in an efrort to idan- 
meny. and correct ossible interface problems. QA 


personnel examine the requirements design and program 
specifications. 


Prelimina and Detailed Design Boe COn alc od 
ELIOT to the Prelim. pa ry DesigaA Ree ea (PDR) and <he 
ritical Design Review (CDRP -lespectively, these 

audits are concerned Magen the ©£OLMae Bhd  conten= of 

desi n deewletca-LOoMeanad Project t6st plans. Tas 
results of these audits are than discussed at “he DDR 
and CDR Beep ee sei 


Independent Quality Audit - Not only do the Product 
Assurance personne aude n> Var toOusS aAzees Of <hs 
software development pro} one but they i: eWeek S 
audited by the corporates Product Assurance organs icles 
= hour This audit does in fact cove ee Wn © > 
project, however, the QA and cM areas are exam= ined in 
etail. Uron complstion of tot audit both the Projece 
Manager and Assistant Project Manager fOr PEeoauec: 
Assurance receive copies Oo PReomeva2 2; PepOns. -Uhe 
also receive any Corrective Action Requests CAR 
Weecr dOCUMeENn= ahy discrepancies Found by the audit. 
Nicene pOees “se lhe. tI INdangs/eesults of the above 
audits are provide 24 +o both the Project Manager an 
“he Assistant Project NWeneg=s fOr the appropriate 
development area. In add==5043 TO ewe L2ndings <econ= 
Menddvtoms eres als> encluded. A periodic summary 
Wiech pde@calisS .ths Saree Oe las ex (0lalal ie perrorned, 
she discrepancies f GOnrecenvs action | in 


progress cr inplemented and a. follow- ~up on correc 
action on=-aqoin from previous reports aS 
prepared by Quality Assurance and provided «oc 
PLOJject Manager. 


t 


re) 
(try 
mons 
‘pO WD 


AS previously noted at the beginning of this 


Subsection reviews are conducted to critique documentation. 


Wnat roliows is a list of the reviews conducted at IRR. A 


Beret description of each review is also provided. 


1. 


Software SCR GEC IIs ee Review -_ This review is 
conducted Been completion of che Software 
Requirements Definition phas2. QA personnel sit as 
members of the review board. The purpose of the 
review is <*o0 insur that ths software requirements 
specifications for the propossd software? project do 
=n fact match the asers opérational requirements for 


the systen. 


Preliminary, Design Review - In addition.*o conducting 
the Preliminary Design Audit, Qualit Assurance 
personnel act as recording secre*ary Lengel s 





b) Requiremntes satisfaction, interface definition 


3. Critical, Design Review - In.addition ‘*o performina 
the Detailed Design Audit Quaiicty Sth personne 
serve te recording, secretary. The enph asis of the CDR 
es on, the ad yaar Fion of required ma ae AS PELeti ng 
content an allocation of each reguiremen= to 4 
functional design slement Osez5 Seale 

4, Design Walkthroughs- These walkthroughs are hald 
Selig lly and oftenduring the D3 sign phasé. The walk- 
shroughs are conducted by techniCal personnel who are 
saken through the design on a stép-by-step basis 
~nfotmally by, the nef see pe) Piss  "dOne 1h ap eerror= 
Omen = Om —he Gons-s.ercy >: 220 gee Ss pemmeset oss clensy 
SouiSuactaon “Of equirements and compieteness 
f Ref. 31%. 

Gee les. Ing 
[Rert. 31] 1% states that Quality Assurance 
has review authority for all test plans and procedures 
S 


Mae w.ated by TRW for 


ce) 


. ’ 


SS 


ct 
cf 


Fh 


Wn 


© 


moet ing and the inspection and surveill 


tests. 
Ve 


erforms a selective review of 


tware requirements. Th2 two prinary fun 


c = 
by Quality Assurance durig the Testing phase are test 
an a 


ale lela forna 


1 


el 
SaeeocronGgmescest PreQcea rss diztest ly correlate with the 


Both these areas are described below. 


Test Audit - This audi Geos .somaueted at the end of 
each test phase and its primary purpose is to: 


a) Assure that software configuration management 
procedures ar= being followed. 


% 


Db) assure pone the test specifications being used by 
the test group 2re the currant approved vérsions. 
c) Assure that test reports identify proper. test 
Reo cSuaTS and software con =qurati HOR: Specity the 
est analysis, and 1f any ficiencies were noted 
how they were explained aie accounted BiOpa. 
d) Verify that tes proc ae ee provide a step-by-step 
Eduroncdle Meser COoNnductang a test and that test 
results comply with te criteria specified 


in the test procedursa 


é) Verif; a oue Test Data Folders comply with approved 
forma 
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ee eecOlveratec Di enc Tes. “ean Wot Ss -aced 
Management vorocejiures fer shange control, _itscre- 
pancy reporting, anda test reporting {[Ref. 30}j- 

2.) VWegepeszion and eve aa Como me Oule yee So Dats 2s 
an on-site activi aes performed we QA personnel during 
program testing. pur poss iS. COs 
a) Monitor all tests to ensure, that the actual tests 

erformed are those specified in the documented 


est procedurés. 


b) Assure all potential discrepancies are recorded in 
the aprroved manner. 


hardware/softwar 
inst the pone tants 
oe 


c) Compare COntI a rac. om 0 
compen+s used in the test 2 
fon went Utied 2p tee Ss2se 


d) See SY thet 3 
the test satis 
acceptable crit 

compiete. 


Hh 
3 ole 


1D 
wunM 
cr 
ts’ 
> (ryt 
(DO tv 
il 
oOMan 
Hm O 
Ma tl4 
pwr 
ry 6D 
wo 
jobs (+ 
W) par 


IDO 
fer 

4 
cra 


v t 


e) Assure that master copies 92 
EPesttemorand Lest Cepores a 
labia rrom a centralized rs 


QQrtrn 
Io-ct 
Qs 
(D UW 
Ion 
ct J4-O 
woaQ 
tM O 
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= 
ict 


ray) 
UWI ct 
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DP Fuss ry 


D 3 (D 


eh) 


O wD 
a | 
c= | uD 
thm NaN 


r~NODNA< 
$v 


Be Lest Fes aie =~ The tese dassc 
OF “3 GO8) ae hical errors : 
tong as & do not deviates 
Ment Ss, Ber hay also make chan al 
and in addition he ee make change 
step sequence provided that t2st pa 
meee eel m2c2 es ta LOmmenang ed. 
documented and will be reviewed for a 
@ese Revicw Board smd @eCOnlo > Juration 

changes causing 2 te Oe Om 
izmements mus*= be submitted through both 
moe Gent EOL Board and the customer 


DOOM than 
e 
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leases =he Vehicle Wwsed 


ange request (TC di 
S + procedures. A descrip- 
2 


uesting change 

£ the problem a 
deen: tne TER. 

s Lotn the sclut 
O 

2 


Serr “TO me 
S-om won 


— cri) 


Dr eos ee changes are 
po DO Oivea ene -CCB = 
oe to the lem and the 


implement in ao lnc lene . ano bia Mam Si ysc 


c 
Iss a satiole (CR 
he Problem Reporting and Review 


Ln addacwonmto “ete “problem ceporting and review 
procedures, the procedures used at TRW for the control of 
changes to the scftware product will also be discussed. The 
Searge control procedures Zit in at this point because mores 
often then not changes ire made in response to problems 

which arise during the development process. 
fee CONEsGULretion Control Board (CGB =) ot. TRW the 


Soni gueade1on Control’ Board 4as been established ts 
review and approve changes to documentation and 


ag 
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oe oe 


TCR No. _ 


TEST CHANGE REQUEST 
PAGE OF 


ORIGINATOR | ) MAILING ADORESS | PHONE DATE 
DOCUMENT TITLE DATE ISSUED 


TCR TYPE: (J ROUTINE C) PRIORITY 
PROBLEM 





PROPOSED CHANGE 


APM SIGNATURE 


CMO DATE RECEIVED TREANITIAL REVIEW DATE TRB ACTION ASSIGNED OUE DATE 


i. F 
a Atos 


Ye 


CLASSIFICATION: BE at RELATED SPR/OPR. 


ACCEPTED CHANGE 








TESTS TO BE RERUN 









("] APPROVED FOR IMPLEMENTATION | SCHEDULED IMPLEMENTATION DATE 
DISPOSITION ["] suBMIT EcP 

[_] REJECT AND CLOSE 
APM SIGNATURE DATE TRB CHAIRMAN SIGNATURE 





Figure 3.6 Test Change Request. 
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So@evdGea  ste0@lariy SeWedmicd meetings are neid but 
ememeaeuroray ices 42 beatae 2s Che Project Memeqe=> can 
Geli @ Special mecting. Fhe project manager acts as 
chairman and members include personnel from Product 
faouLeanGS., Data gear TO ces >c2.2.¢0 Syectem= Ee Ggenecrang, 
Meco NON SOLE Was em OyScemS ana Support Software 
Data Processing Hardware, and Integration an 
Testing. The user may also take ee in the meetings 
and inany event will be supplied a copy of the 
Minutes. 

The following are the documents used at TRW in 
an effort to mairtain configuration control over a software 
development proje¢ct. These documents are submitted to the 
CCB for its review and approval. 

1. Design Problem Report (DPR). - This _reporz is_used to 
request changes to baselined (already approved) docu- 
Tae eC (ateO ts) 2ntt te ceechanges fo formaliy 
has¢iined customer controilsi documents. The DER 
Semedins (~SO2i a dSseription of MeenS problem and a 
peo see solution. Once it has been approved by tne 

Ontiguration Control Board if provides *he accépted 
soluzion to be impiemented. Marked up chang? pages 
(usc cmdcsachied ta) the DPRMeoe tus crate the changes 
being made, QA et iabeR LS tepe-Or the resolutzen of 
ais DPRS [Ret. 32). beeiteowees! )6©6[ REEL. 32] is an 
example of the DPR used at IRW. 

2. Scftware Froblem Report (SPR) - The SPR is_a request 
poe peo some le wehany io. cOsmmeiis Dally controlled code. 
em MeulGe= a -ccSC pe ommos sn= problem Pct... LSS 
Suc rii pr ary end. ="SUzemes affocted and Bau22s2 a 
problem soiu*ion. Ince approved by the [CB it serves 
Gewee (2 tner 7ae Oo come mpasate the master library 
Ot ae Ss We Pl2gUGcmmewomeners 32] iS an exemple of 

RW's Software Problem Report. 

eee Lemporary Modsfitcation Notice (THN) - The TMN 
Begwests ana am Fe one a aa esr enangjes 20 base= 
meen code. I(nCludc@mw: th =e TMN are a listing of 
Hemi mecmanges, Teaseons fOr Teo. change, any cestric- 
PeEoMs sc cstl ng, Veuz ticatlon and riles affected. The 
TMNS are correlated to SPRs which implement the 
permanent change [Ref. 32]. Figure 3.9 Ret. S32 2s 
an example of a Temporary Modification Notice. 

fee Penetics 

Miseme© acc sches NO=a2S in  —[Ref. 30] that TRW 
received the following benefits through the implementation 
of its QA Progran: 

7 it has PE U ee increased management visibilty into 
the development process through reviews and audits. 

DP Sees risk has been reduced through better require- 
Mente omenaccasstity and more disciplined and thorough 
testing. 

3. It has enforced software standards. 


V2 





TR W/ DPR No. 


DESIGN PROBLEM REPORT PAGE OF 
ORGANIZATION / MAILING AOORESS DATE 





ORIGINATORS LAST NAME INITIALS 





DOCUMENT TITLE A REV, DATE 
OTHER ITEMS AFFECTED (CPCis, INTERFACE DOCUMENTS ETC.) REFERENCE PROBLEM REPORTS 


PROBLEM DESCRIPTION 


» , 
x " gy 
APM SIGNATURE DATE 
PROPOSED SOLUTION > * 
egy ~eaty 
“Pe 
a 
‘ >. yy 
A, 7 
Py a5 7 Thktppenpy 
* ga 
; : 
Le SS 2 ? apm SIGNATURE Ju COATE 
ccs INITIAL REV: DA CCB ACTION ASSIGNED ACTION OUE DATE 





ae Tat “s 


CMO DATE RECEIVED 


CLASSIFICATION: AFFECTS FORM, FIT, FUNCTION, COST OR SCHEDULE O YES C) NO 
ACCEPTED SOLUTION 


FINAL (J APPROVED FOR IMPLEMENTATION SCHEDULED IMPLEMENTATION DATE 


Disposition © SUBMIT ECP NUMBER 
REJECT AND CLOSE DPR 


CCB CHAIRMAN SIGNATURE DATE CLOSED 





Figure 3.7 Design Problem Report. 
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YW. The development and maintenance orf software tools has 
been centralized. 

o- Quality, Assurance records have been centralized. 
These include, PLEObiem Teports, deviations and 
waaiyvers, reviews and audits, and test and insosection 
reports among others. 

6s. A Skill center for personnel with multi-project visi- 


& 
bility whe are better able to prepare roject pians 
and procedures. 


7. An independent group assuring that deliverabie items 
meet contractual requirements. 


j.- Lessens Learned 


co 
Ome? 


cr 


rr) 


Kurt Fischer states, Upeenably the larg 
lesson learned is that ona key to the successful development 
Meee scortweare is the employment of a strong QA activity", 
(Ref. 30]. iid One cho nape yen re sche: DOinzcsS our in 

O 


fRef. 30] that ‘the following Lessons were learned by TRW 


0 
during the implementation of its QA Progran: 

1. Insure adequate QA, participation duting the proposal 
emicONteaeGe GAElinition phases. 

2. Hiring versonnel knowleddqe2bl2 in software and then 
Srecieng | onem 20 pr been assurance is easier than 
hiring QA people and training them in software. 

eS Onl tne. f bac amass,  Serly -2n the developnen* 
BeOcess~O allow plenty of time for cofrectave action. 

4. Announce the audit well before it taxes place. The 
Cigiece —soetO, CHSUGsme = WaS done Didht, Net <0 far 
problems. 

PEVOtiocower an saAuUgl= Sheckiise and distribute it +o the 
area being audit2i, at the same time the audit <is 
announced. This eliminates subjective assessments by 
Picea die ton and dNSQENS tne par ty being audited of 


the exact scope and depth 
Oo ener aud t. 


6. ASSiaqn QA engineers, on along tern 
may develop a4 relationship 
with each project sub-group. Ohw = 
colocate with the iavelopmert 
personnel so that they might bet 
problems of other project members. 
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TEST ACTIVITY__.__ C—O‘ ‘(‘( MCT ESTCCASELNUUULULLCCC( O™”CRNN TATE 
7 CCIE LOCA TION ee LT SC NAPACT CODE 
oO 
us| S/W TYPE: T] DATABASE FILE 10 ELEMENT 
f- 2) 
= (J os PROGRAM/VERSION CPCI# 

[] APPLICATION ROUTINE/VERSION 
Cl support TEST PROCEOURE NO. REV 
DESCRIPTION: Vy 
~. Py 
= . a *y 
= oe ws nH me. » 
2 oN ry To 
2 
fo 4 ™ 7, 
a ~~ ‘ 
— 
Se. ~ 4 
. —, 

MANAGER SIGNATURE AUTH. TO SUBMIT. 9 rn, oe DATE 

DATE ADDRESSED. PRIORITY CODE” CHECK BOX 
| an md iF PROBLEM 
z | ACCEPT — ASSIGNEE _" QUE DATE 
= oven = DEVIATION | 
UO REJECT - REASON WAIVER NO, | 
< , : l 
xo DEFER - UNTIL | 
o 

AUTH. SIGN. TO PROCEED IS A 


DISPOSITION 


SOFTWARE PROBLEM REPORT 


PROJECT: 


NAME ; 


























AUTH. SIGN, TO IMPLEMENT FIX 


CERTIFICATION @ ANALYST: WU CDATE @ MANAGER: 





DEFECT CATEGORY... C—“‘CCOPROGG/V'SNN AF FECTEDO 
ROUTINE(S)WSN AFFECTED 
DOCUMENTS AFFECTED (TITLE/REV/DATE): 





BOCA TION 2 ee. 6 PHONE: 








SPR # 


Page } of 


Felt mae Ae ON eos OATE PREPARED: 








DISCREPANCY 


DATE 











FIX DESCRIPTION: 














PP ATTACH LISTING OF EXACT CODE CHANGES @ SCO NO. 


eee ae ae eee Oe 





Figure 3.8 Software Problem Report. 


US 











CHECK IF 
STATUS IS 
"CLOSED 











_ a as Se ce aqmeawaw ==) —_ a a ae ae oe 


ae Background 


The Naval Ocean Systems Center (NOSC) esta 
lished its software Quality Assurance progzam to 
Beocuring activities in aSquiting quality software. Q 
software is defined as software which neets all requiremensi 
Se Operabrlity, Pelsability, and Maint daamability. The 


Le 


t? 


expressed mission of the Software Quality Control organ 
fon 2S *©0O provide assistance to projec* managers in the 
+ 


acguisition/management of higher quality software products 


‘o) 
rh 
QQ 
i) 
iq 
wir 
ry) 


tercuch the implementation 
provides a manageabie structure to the software development 
PaecessS cthroughedocument iLnspectior, configuration manage- 
men+, ideacesteig. Suppor t. Standard techniques includes 
Miepecting and evaluating the documentation of computer 
systems, evaluating a projects configuraticnh management 
procedures in regards to software documentation and ccmputer 
of tested programs zhrough 


qs ie 


Beograms, assuring «he int 


ty 


<= 


y 
faeegiem .ibrary control, increase user confidence «hrough 
a 


Meseeng, and being an active op 


ty 
ct 
} 
Q) 
}+ 
ue) 


ame sOhe Dre jeer Changs 
Men<cloOi Boards. 

MicweGetvec es Of the SoEEWare OUality Control 
Organization are geared *o the projects life-cycle. This is 
mewe whetner the project is short, requirinag a mininun 
effort, or long, extending over a multi-year veriod. Because 
Mmme nis fact €ach SQC st?p is tiei to the project plan, 
milestone schedule, and list of configuration items expected 
to be baselined [Ref. 331}. 

At NCSC the Quality Assurance office has been 
established at the directorate level. Figure 3.10 (Ref. 33] 


shows the NOSC organizational structure. 
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be. Quality Assurance Dbjectives and Policies 


The following are the mos* significant of the 
objectives established for the Software Quality Assuranc 
Sean ization: 

1. Ensure consistency 2f software baseline development. 
2e Ensure corpiiance with design standards. 

3. Ensure compliance with programming standards. 

4. Ensure adequacy and completeness of testing. 

5. Review all test results. [Ref. 33] 
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The folicwing policies have been established at 
regards to Software Quality Assurance: 

Pie maeve scpeetG wor Serore.c 125 che besic _responsi- 
pee he Pn eg Ve Orme rOlueGts delavered by NOSC. 
Each Director shall utilize the established quality 
assurance resources, as appropriate, to assur? 
adequate quality of all end products. 
toe oars Oa seNemmeng reeling and Computer Sc! erces 
directorate acts as Center managements agent for 


product quality assurance on all Center projects. As 
such he will eep Center management infsrmed of the 
results of reviews and audits c£& products developed 
and produced by the center. 


The Quality Assurance office, which reports directly 
men, ene Director ~of the. Peles cictee Cle and Computer 
Sciences Cilrectorate, Wowimege tie POlhe 9 Of COncact 
EOE the Cecordindtion of all Center guality assurance 
activities. The OA office is responsible For keeping 
he Center's QA policies current and cfcesponsive t9 
higher Navy directives and instructions. [{Ref. 33] 


Cc. Quality Assurance Pianning 


DMiewsOrtware Ouality Con=fol Plan is prepared by 


See personnel through intetviews conducted with the Project 


Manager. The plan is csordinated with project plans and 


Bagveges a description of how the elements of quality 


Teameagement willl te applied to the project. Each SOC plan is 


tailored to project requirements. 


be 
tive the SQC plan must contain certain elements. These are 
qm 


planning for products at the end of each task usin 


(D 


ee 


It 
T 


For the Quality Assurans2 proacram to e- 


anage- 


ment audits and reviews as a measur2 of completion of the 


products, developing documentation in the proper sequence, 


and 


accepting SQC inspection assistance to the Project 


Manager as an aid in ensuring successful system turnover 


inet. 


Sible for developing and enforcing design and programni 
W 


standards. In the area of design, standards dealing 


33}. 


d. Software Standards 


Software Quality Control personnel are respon- 
a6 


Nn 


1.Q 


Lael 


iis, 





ames ecicea se logical SEficirency, =sechnical maturity, and 
consistency with functional specifications must be enforced. 

Programming standards orovide th: required 
consistency in the technique and processing required for 
continued software support throughsut the system's Jlife- 
cycle. The programming standards ical with the following 
areas: logic and coding tonventions, flow chart standards, 
intermodule communications, programming language structure 
and use, data design, module segmentation, and logic error 
checking. 

Quality Assurance personnel review the progranm- 
mena effort with regard for compliance with standards. tf: 
any non-compliance is found they will take steps *o ensure 


conformity with the standards. 
e. Quality Assurance Reviews and Audits 


The software review process exists so that 2 


qualified decisicn may be made to resscmmend advancemen+ from 


one phase +o the next. Adee Sia ciecwoene= Shand fare 
S9pauctea to verify configuration items conform <t9 specifi-~ 
Cations and other contract requirements. The results of 


Ss 
reviews and audits are reported directly to the development 
directorate by quality assurance personne 

In addition to the craviews and audits the 
process of baselining will be discussed. It is included here 
because it is an integral part of the audit and review 
process. 

Baselines are a configuration management tech- 
Nique used to control the development cf a software product. 
as Stated in [Ref. 34], 


A projects software configuration is the prevailing 
Szate or its softwar 2 components. Those components 
which are subjected t) systematic management are termed 
Configuration Items (CIs) .,.Software CIs are qualified as 
meng eelements Of en evolving software product which are 
set BObe wie in technical docunentation (on clung 
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Mec aae scons, ebawengs, and Jistings), and achieved in 
Sire compu csr program i7~s2li (rcresiden= on card, tape, or 
cl acing an 


fief Ref. 33] it goes on to point out, 


Baselines are employed throughout. the life-cycle of a 
configuration item to ensure Orderly, transition from one 
Rajor commitment point to the next in. the system engi- 
neéring, software Sec DEOGMGweon, sand Logistic 
Support processes. Baselines are estaplished at those 
goents in a program wh2re it is necessary to define 

Solcieaeparture DOInts ECOG conttol of future changes in 
performance, | design, ievelopment, Decalece On, and 
related technical requirements. 


TEa dase 


_— ~ 


« Zs & vas 4 re = 
mei eS eaceeen i cConmaecceotance Of Aa 


ilu 
Pp 
< 
a) 

f+ 
O 
rt 
3 
‘Dp 
3 
ie 


@ecument OF product. 
There are four types of baselines used at NOSC, 
mime: lone, Allocated, Product, and Iperational [Ref. 34}. 


peo=26l description of each follows. 


fee eune=.Onale Baseline = Th2s considered the highest 
level baseline. The technical documentation at this 
level delineates all the age sce Eunctional charac- 
teristics, the tssts which will be required to 
demonstrate their achievement, all necessary inter- 
TACES, ae. “GRVe aneeee ds ec NuwCOUStma2n Ss. This 
baseiine generally covers See IO CUnGN tacson 
produced prer to development of the software design 
and the formal System Design Review. 


2. Ailocated Baseline - At this the _nex= lower 
baseline, the SrEecimance oriented specific 
Moen Aae- | SUbOrathate to the Conitquration I 
mieewtuneciona il evel, expand upon allocated func- 
Sema l —naracteristics. All documentation op 
Short of the Preliminary Desijyn Review is cov 
this baseline. 


considered the lowes* 
is DOLME Which is 
feomai an ailocated 
Sveosacwon, adi. =— 
SseOr its iife-cyele. 
Dea One sand DEOgrams 
ification review. 


3. Product RPaseline - T 
ievel. The documenta 
subordinate to bot 
feyels, detines the 
Mem Cee, and JOgistic 
Tees ornally covers 
Paoaducea Pr=0r tomEehe F 


4. Operational Baseline .- Once <= 
passes the Formal Qualification 
~~ meets operational requiremen 
baseline is established. All a 
to the system during its life-cy 
P=OMechaTs baseline. 


developed system 
le 


wand has proved 
the operational 
fications required 
will be performed 
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Ths 


beselining technigue is also 


TRA during its 


fo 
(fh 
iD 
fs 
us" 
hey 


software deveiopment process. 


BOl@meweng 25 2 J2ske or the reviews conducted by 
mee SOL*ware Quality Control organization at NOSC. A brief 
description of eacn review is included. 

1. Initiation Review - The purpose of this review is to 
atfirm. the Ses ae CO ae Requirement as the basic 
quideline for the project. Prior to the review the 
Operational Requirement should have been read bY sh dual 
members ci the project tam enn software 

wast y assurance ersonnel. The Jperational 
‘equizement is reviewed to ensure no reguirements or 
constraints have been omitted (Ref. 33]. 

oe ee ee Review -_ The objective _of this 
review 1s to determine the alequacy of the develop- 
ee Sefiormes in dsfining System requirsmerts. _ The 
Mew eo CONGUeGL ca Once 4 SaJhi Sican= porzicr of fae 
ie functional requirements have beén established 
{ref. 33]. 

3. Document Feview and Substantiation - This occurs in 
€ach phase of the adsvelopment Lite-cycle. The primary 
CEMmGern 25 to. review ies cc SSOCI Mera ct. ON EOE 
compieteness and correctness. . Quaiity assurance 
Dessonge  Nepece and Substantiate the conten= of 
ail documentation. The documentation under review is 
compared against established standards SOren Sime < 
eermmeas ns tne DeOopee Toye of detail and that all tae 
moqweewed "GoNuony 88s UrseSen>. NOt only is a singles 
Paec=e of decimnemeact on reviewed by itself but it is 
Colpemcdmecou ail aSsOeGmatea GZocumentetion to ensure 
completeness and consistency. All deviations from 
Standards and ay we Giawect ee PrODLemMS, are noted and 
SoNemccdmeO DOr ene Pregese “Manager and the devel- 
Gpeberor COEnSECCEION. Once the Operational Requirement 
has been reviewed and aporoveji, the remainder or the 
documentation produced py the project is reviswed to 
Sesire lt 25 Comsistent with and meezs the require- 
ments set forth i the Jperational Requirement 
[ Ref. 33]. 

4. System Design Review - Once the project team has 
determined that the software requitements documents 
Fulfill all requirements and present a suitable allo- 
cation of performance requirsments between hardware, 
software, and human actions, th2 System Design Review 
1s held. Also the identification, Correlation, 
completeness, and risk of the software requirements 
documents is evaluated. Upon SD SECU EN: these documents 
are considered baselined PRef. 5) 416 

SB. Prelimina Design Review (PDR) - The prinarv 
concerns of =the PDR are the software design and the 
completion. of requirements set forth in previousi 
baselined documents. The itens which are considere 
from the program design documents include: 
couple ce progam fLunctaonal flow charts. 


b) Storage allocation charts. 


C\eecemesed 2unerti ona l descriptions 
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d) Structure and organization of the data base. 
€) Functional interfaces. [Ref. 33] 


6. Critical [esign Reviews - These reviews are conducted 
as individual system programs or modules are speci- 
fied. The primary concefIns here are that required 
Standards are met, all prior baselined functions are 
Ei Itedweand that Prlegr to coding, the lowest leyel 
of design detai has been reached. The following 
items (Ref. 33] are under consideration from the 
program specifications: 


a) Compatability of design with functional inter- 
faces. 
b) Data base in Gevon'S< 


tera 
grity Of Poqecasdtaqecans, algorithns, 
OGac: 


Geo esign ant 
al Eons, and f£how chars. 


GWeeealtdwesce interfaces. 


€) Human interfaces. 

Meee rGimal Qualification Review - This is conducted upon 
Coulee Cmmor Born tne Funccional Configuration Audi= 
SUS eno Gal CONnteguration Audit and .arter all 
GesccLrepalh cles uUnce Vered _ these audits are 
corrected. The purpos2? here is to ensure that the 
system as developed and testei meets all requitements 


So eee ern tieene Ob=statironel Requizemen=s. From this 
review it is determined that: 


ay User requirements have been satisfied. 
Ces 


b) Documentation is, sufficiant ct 
PawOugn@ut, {ts 12 fe-eyels. 


Cumvico-SerUNGtlOns ane aicquac2=iy described and docu- 
mented . 


&) Testing is sufficient te ensure user confidence. 
e 


e) All outstanding deficiency reports have been 
resolved. 


f) The outcome of the review will either be a4 
successful completion or the determination that 
further development is necessary [Ref. 33}. 

8. Post Development Reviews - These reviews are 
conducted as_ the operational evaluations of the 
system are made. The focus is _on Stegss development 
PacbieMeaceaS, Operatztonal ditficuities, and unde- 


tected deficiencies [Ref. 33]. 
The following list contains the major audits 


conducted by the NOSC Software Quality Assurance group 


during the software development process. As was the case 
wit the reviews,,. abri2f description of each audit is 
included. 
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(ees emcees eo rgund = One AIdL= = This serves as the 
primary means or YS ae eer that deveiopment of 4 
software Configuration [tem has been completed satis- 
EQe wer ily. Once 2, document has one threugh *hea 
Pee eames ec Omenaocess (described ecatlies} 
QueeecOrdme: the Geyview forms and diSposicion of defi- 
ciencies is maintained. The apdated document and all 
deficiency reports are Laputs co chis audit 


(Ref. 33]. 


Pee Uineccelona | Comittglra trons Audit -._Phis 1S a critical 
GOtpee son Ob al tee les —cest/anaiysis data and func- 
tionel specifications to validate that the item as 
designed and developed, meets all the functional 
performance requirements specified in its development 
Specificazion [ Ret. 35 }. 

3. Physical Configuration Audit - This iS_ a comparison 
of the "as buiit" item with its approved and released 
technical documentation. Ths objective is to ensure 
het the documentatton iS complete and iS appropriate 
E0rD operational maintenance and support purvooses 
[ ker. 35]. - 


BOtimeene Lime: HOmats GOnr  gquzatitonm Audit and the 
femesceca! SContagiracion Audit are conducted prior to the 
mame sSiOn Cl a Configuration item for ‘formal acceptance. 
From [Ref. 33] the purpos2= of these audits is: 

Weecotai= Mm CCMDLsence to Change contrel vrocedurss and 
tO 6ensure Only approved changes have been imple- 
mented. 

Pees DeEwensure that objestives are sufficien:. 

3. TO ensure the developed softw 
as the specified software pro 

=. Testing 


Software test and evaluation, as conducted by 
the Software Quality Assurance personnel, is designed to 
ensure that che software, as déveloped, mests the original 
requirements Ser Ont h by <the user/sponsor in the 
Operational Requirement ani that it performs as defined in 
the system documentation. By conducting an analysis, tech- 
hical evaluation, and a detailed review of project 
documentation the necessary inputs to prepare test plans, 
test specifications, and test procedures, are obtained. 
These documents are then used in the actual testing and 
evaluation of test results to verify that the system, as 
mee rOoped smeces all technical specifications [{Ref. 33]. 
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end aCtiv=t-es, A brief discussion of each is included. 

Leste. een ais LSwewELctsni by SOLtware Quality 
Control personnel +9 definr the 

scope of tests required to assure that the softwares 
meets all required specifications, Al*hough the test 
plan is a high lev2l document, from_ which the test 
Sece Selec ons mec Wateeen, te Still must identify 
tne degree of owas Phere opeclt i Meme tunctions t5 
be tested. The schedule for individual tests and 3 
Summary of the environment to be used are also 
included in the test plan. The plan is reviewed by 
the software developers and approved by xhe 
Program/Project Manager [Ref. 36]. 

Mest Speclrications . = Software Cie cy CON Go| 
prepares a test specification for each test ir the 
Boot Plat « Based On requirenents set forth in «he 
design documentation of the develoning system, the 
Boe secu cacloumede:, Nes she  DaSic 2SSt Caregsris 
and the general mnethoas to be used in a_ specific 
peso, ONGCe a test Specification is prepaced 1= forms 
the basis for the development of test procedures. [In 
Aegt en -Oeaecrinong che Scop> Of the Specific tescs 
ese SPpeCizications S.ate the purpose or he zest and 
identify the software, hardware, and/or system =o be 
Pomme. Meinere  MUSst be  Sutfticient intotmation in 3 
Pesto Specteteat on sO that test procedures may »b¢ 
developed and so the results of defined tests may be 
evaluated. These are also reviewed by the software 
cevelopers and approved by the Progren/Project 
Manager [{ Ref. 36]. 
Test Procedures - These procedures azr2 devaloped by 
ome oom Nason Ous le =) Contes! “personnel using es< 
Be ee eee ONS, | ioe tS iene! S, and cuhe= >elevan* 
Cesign documentation. The pea purpose of che test 
rocedures is to present SSo eed Sth uUCc=LOnNS FOr 
Gene sess CXeECULIONn and the evaluation of test 
results. mhe wee gant 22 teen an structure of the 
processes are expressed in general terms in the *+es* 
procedure aleng with GONSera Nis woe aSSUMmpTc2ONsS 
~mposed on their usage. A description of the total 
equipment, manpower, computer program, and supporting 
documenta*ion rejuired for operation is also 
provided, All hardware or software revisions or modi- 
rFications must be specified along with any required 
pre-test checkout required t9 ensure a valid tes* 


environment [{ Ref. 36]. 


Software Testing - Software Quality Control. personnel 
may perform testing in either of two anvironments: 
Sameaced or "on-site, In a. Simulated environmen= 
Gereain subsystems, iinks, and peripherals are avai- 
Tage it Or Le eCoaess PilesOSeH OL PpEaaduccion and 
Sant o: BO ie Sacer tate 1s done uSing the system 
“-n real-life environment. The policies and procedures 
are set forth _in the test ent specifications, and 


roceduresused in accomplishing software testing 
Ref. 36]. 
Bemenemcet Ss = AS Cach test is completed a Test 
Repos $LS WEttLten to document the satisfactory or 


Uned-isctactOnry completion of the test. Any and all 
deviations from test procedures or equipment mnalfunc- 
tions must also appear on the Test Report. Each 
apparent system discrepancy will be noted by the 
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Sentrol 


submissior of a seperate incident report ‘ Die Test 
Re Poe twee ie Set -2on ce ail completed tes® orcececure 

=eps to allow programmers to duplicate the condi- 
o ons in which any apparent incident was discovered. 
All completed Test REPOLTS Become a@ part cf «he 


permanen system documentation { Ref. 36]. 
ge Problem Reporting and Reviews 


As was the case with IRW the area of change 


cs also discussed tae chis Sipsect ion. The 


Configuration Control Board and the documents used in 


configuration cecntrol are discussed Im 06 tee Last 6that 


follows. 


1. 


Coniero 4 —- Tne DUS pose wee = 
control is to manage and monitor 
40ns *>  baselined configuration 

PO user, software developer, Or any 
other member of th? project dsrganization may voropose 
Changes <c the sottware. Theses, along with any prob- 
lems uncovered during aoe and evaluation, are 
eC rastaea to the SOA oe Naa GOPecolL BOoara (CCB) as 

ther pa oa eee Chan =oposals, deviations 
waivers. he in teen investigates all Senge 
requests, Acie es and waivers and based on docu- 
mented analysis, recommends Rosanesee aC OT 
(Ref. S20) ieee ce:3 05 ads up, of representatives 
ipeom e2: project ar focked. activi iss. Tie 2 sence. 
Manager is tne chairman and. tie Quality Assurance 
representative acts as recording secretary. Although 
*he final decision. regartas okay Si ak eugiios OE eo oapebe ily, 
rests with the chairnan, NeSwseOlsZeits  €xXDery acve ee 
==om Pucncc. parts Gleancs 8 such a$ Softwares 
Development, Systems pcg Gi, QYalsty Cong=son, 
<1 3 
e 


t+ 


Logistics es eta User, and 
Facilities/Hardware managemen 


1H) wD 


Engineering Change Proposal - Used in _submictting 
pro osed_ ” changes to baselinad software configuration. 
Either DD Form 1692 or 1693 is used [Ref. 34]. 


Request for Deviation - This is used when it is 
necessary to depart te eae zly from documented 
requirements. DD Form LS used in this case 
[Ref. 34]. 


Request for Waiver. - If.an 
*ts required configuration 
development error, 4a Request 
This is also submitted on a 


| 4 


, 
OOrhyw ctr 


we VCOneern ce 
Wom Gue 20. 3 
foes Pouce. =o 
94 [ Ref. 34}. 


OH tb 


Engi BOGE Dig Change. Ordeczs - 
granted, Paes Change 
and Waivers are implemented 
Change Order [Ref. 34]. 


: ecification Chang shone Melo Ie ee ce oe Revision - Both 

ese documents apa used when an Engineering Change 
proposal ore Waiver affects baselined documents. or 
eet de FeENesvOlG sis Used. EOE the SCN and DD 
Form 1695 Ae Bees for the NOR [Ref. 348}. 


Ss OU ba Ss been 
osals, Deviations, 
ugh an Engineering 


Lom me eo tat Cus 


cr'gO 
HOoOQ O 
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feo newate Teouple/incident Report - This_is used for 
Oc ) OST aS Balm “SSS PrOCSautes, Sgui0- 
ment malfunctions and software anamolies [Ref. 36]. 
Dimes Become a Past Of the soteware Test Report and a 
permanent part or system docunentation. 


Eee bP EOg EallealLepraty Cone =o 1 


Program Library Control is designed to maintain 
and control a systems computer programs and all changes to 
these programs. Software Quality Assurance personnel ars 
responsible for approving all changes to Jlibrary programs. 
This system not cnly provides baseline error-free, patch 
free programs fcr test and Wrecasr but i2 also insures 
meatal. approved Changes 2re incorporated into the programs 


n 
fRef. 37]. TRW uses a system similar to this. 
i. Benefits 


In {[ Ref. 33] it states that NO 
Sie <OliOWing benefits through <hs estabiishn 


Software Quality Assuranc= prograam: 


femeic SStablishes a controllabis  e=ructure in the sof<- 
ware development process. 

2. It assures certain elements of quality in every phase 
or the develovment. 

Sey -calci ng sework Substenttaliy, i. provides for =he 
satisfaction of requirements. 

wee ine SUDSz antial reduction an the amoun= of rework has 
lead to a Significant savings in life-cycle costs. 


3. General Electric Company 


Sent TS Cauc tion 


oO 


Tha software quality assurance activities at tw 
separate divisions of the General Elactric Company will be 
discussed. One discussion will cover the Electronic Systems 
Division located in Syracuse, New York. The other will cover 
maemo pace Divisicn located in King of Prussia, Pennsylvania. 


Neither discussicn will b2 extensive 2s both divisions use 
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quality assurance techniques which are similar to the «Ho 


organizations previously described. 
be. Electronic Systems Division 


(1). Quality Assurance Objectives. John 
Pemissick Jr. and Robert A. Price point out in [ Ref. 38) the 
objective of the Computer Software Quality Assurance (CSQA) 
Program at the G. E. Electronic Systems Division is to 
ersure that software delivered unfer a contract meets the 
requirements of the contract. As can be seen this is similar 
to the organizations previously discussed. 

(2) eee ae yessumenes Yanna. EOS ete Gi chs 
of Computer Software Reliability and Quality Assurance and 
d «echnical specialists ate responsible fo 
ementing the QA Program. In addition to providing 
pport to project managers the manager of Computer 
Reliability and Quality Assurance report 

S 


Hemeqe: @m nxellebility ard  Quali« 


vt 


Ss 
A a 

Should be noted th Shere oOnvl ce SSf2 Wale ° Rel tabi 

up Pp 


a 
Ouality Assurance gro LS Osdan* Zar: oOnelly *2nde 
Software Sngineerina. 

After reviewing both the Computer Pregran 
Management Plan and the Software Standards and Procedures 
Manual, Quality aAssuranc2 personnel develop the Computer 
Software Quality Assurance Plan [{Ref. 38]. The plan defines 
all activities which control and assure computer software 
quality. The plan is developed using Mi1-S-52779A, and it 


identifies the organizational rcompon2nt= responsible for each 
ic 


eer avity. 

. (ieee scOftwars Standards. ~Similar to TRW the G. 
Be Electronic Systems Division also develops a Software 
Standards and Procedur2s Manual. This document, which is 


primariiy used by the programming teams, establishes rules, 


@madelines, and limitations which 222 to be observed in 
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generating sceftware designs and coie which will have che 
Maoperties «wits CONSISt2Nncy, readability, and quaiiczyv 
Aci. 3S je Dwee sosilerture anemecOltens of “fhe Sottware 


Development Notebook (SDN) is also d2afined. 

The SDN is similar to the Unit Development 
Folder used by TRW. Tt is a simple lsose-leaf notebook which 
is established during the Preliminary Design phase for each 
Computer Program Component (module). I* provides a common 
Gemtection point for ali informatzon dealing with a CPC and 
it is maintained and updated throughout the remaining phases 
of the scftware development [Ref. 38]. 


The SDN is prokéen dewn into the foliowing 


Bee eens: Requirements, Detail Desian, Functional 
Saopabilicties, Gaiters VescemcCase Descuipemons, Tes: @ Case 
Results, Software Problen Roo reS, and Miscellaneous 


Mieormaction. With the exception of Softwa 
and Miscellaneous Information, each section ccontains a cove 
See. Shewing schedule daztss, actual comp 


eview and approval signatures {Ref. 38]. 


| 


ihe SD, lixe IRw's UDF, serves as the 
Daeure=ple working document of =he programmer. In 
provides management, tne customer, and QA pers 
Pemibey =NtO the desian, status, and quality of «he ¢ 
under developmen* [Ref. 38]. 

SDNs are audited monthly. This audit may 
be on either an announced or unannounced basis. 

(4). Quality Assurance Réviews and pe Aug a.) 

Audits and reviews similar +o those conducted by both TRW 
emaeeNOSC are conducted 32t the G. &. EPlecer onc SyStens 
Division. Internal reviaws.are conducted by software and 
Systems engineers who nave not contributed t9 the design 
e review chairman is réesvonsible for impie- 


i 
under review. T 
menting “he re 

h 


h 
view plan which was approved by Qualit 
= 


Assurance and ¢ Project Manager. 
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Meow eeeomad key OWS are conducted, 
JOLrt customer/cecntractor frevicws ar These reviews 


neid. 
71 


fal 
utuas 


~ 


Ww 
Beov-ce a technical forum for better 


m WndSsa= sak targ OF 
the performance requirements allocated to the computer scft- 
ware, and of the design approach selected +9 meet theses 


requirements (Ref. 38]. 

(ji festang., At the Gs E. Electronic Systens 
Division the Test Plan is developed during the preliminar 
design phase of the development process. Test proceaures 
are developed during the detailed design phase. Actual 
testing cccurs in the final four phases of the development 


process: Cede, Debug ani Unit Test, Develooment Testing, 
[ireedgra=ion Testing, and Acceptance Tasting [Ref. 38]. 

UMesemecest= ng 26 corpndue-ed of individual 
mee ™neS ~O Lreveal coding ¢rreors, computationel errors, 
improper input handling, inappropriate error messages, and 
meonrrec- formatting and content of output. Development 
testing takes the prviously tested routines and combines 
meen cO LOmm Comtpumer Program Componsnts (CPC). fhe suite 


Berai capabilitiss of the cCPrc i mone ole never ete d 


(Q 
as 


Sc ana which is performed by an independent 
test team, combines CPCs and verifies the correct sequencing 
of components, compatible component interfaces, and proper 
Gata routing. Acceptance testing, which is also done by an 
independent team, verifies system level functional require- 
ments. Mesceincludes overall timing ard the abiiity +o 
handle the total input load. As at TRW and NOSC QA personnel 
witness all testing [Ref. 38]. 

(6). Preblem Reporting 2nd Review. jal). leone. 
meweand NOSC, G. Ea Gilectronte Systems Division has 2 
Configuration Control Board which reviews and approves all 
changes to the software. As in both the previously described 
organizations, Quality Assurance personnel are members of 


this board. Quality Assurance personnel are also responsible 
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Zor V 
mBaeed os fh is iS Similar to both NOST and TRW. 

A Software Problem Report (SPR) defines 
h 


hie 
it occured. Quality Assurance receives a copy of all SP&s. 


erifying thakt all appreved changes have been incorpo- 


and documents a problem and the tast conditions under w 


In addition they maintain 2 listing of all outstanding SPRs 
and the party responsible for corrective action. Once a 
problem has been corrected Quality Assurance annotates its 
copy of the SPR with tha way in which the problem was 
corrected [Ref. 38]. 

| Rll coding changes are authorized using a 
Software Change Order (SCO). Chang2s to hardware and soft- 
ware specificaticns are documented on @ Specification Change 
Notice (SCN) [Ref 38}. 


(7). Benefits. An effective Quality Assurance 
Program allows the G. E. Electronic Systems Division to 
deiiver computer software which meets all contractual 


Beguerements. In addition it provid=s management visibiiic' 


MD 


into the software development process. 


DLV 2S eon 


D 


ey sopac 


(1j= Quaiicy Assutance Jbleceives. The Quaiizy 
Assurance Programt which was reviewei has been in effect at 
S 


Ss 
the General Electric Company' Sepevis@en since. -e 19738. 
5c mn 


The primary objective of a 2s to Ensure zhact 
delivered software meets 211 contractual requirements. A 
secondary objective, Which is desigqned to help secure 
eem-racts, is +o define and i ent specific measures 


designed +o ensure delivered g¢ @e incorporates the 
features necessary to achi2ve testability, maintainability, 
memrab a iity, etc. fRef. 29 }. 

(2). Quality Assurance Planning. hs 2s 2 


case in the three previously described organizations the 


primary job of the Quality Assurance group is to prepare <he 


on 





QA Plan. Once the QA Plan has been developed Quality 
Assurance turns its attention +0 the implementation and 


Managemen* orf the plan 


(3). So ie Seandaris- Fach programmer 
working on a given project receives a Copy Of. sche 
Programming Standards Document (PSD), which outlines the 
standards to be used to produce 2 high-qualit software 


product. It is not enough however that the programmer be 
given the PSD and left to go along his merry way. Training 
sessions, conducted by the Software Development group, are 
heid to explain «she content of the PSD. It is the job of the 
Quality Assurance group tover:fy that each programmer 
Peme=Ccipates in the training sessions [R8ef. 29]. 

(4). Quality Assurance Reviews and Audits. The 
Mum spec= D2Vvision conducts revisws and audits similar +9 
those of the previous three organizaticns. They do, however, 


give special attention +o the developmentof scftware inter- 


races. The development of *he software interfaces is done 
by the Software Development Group. However the System 
Engineering group *sS responsible BOD) devs i Opi ng =he 
mmeesrace Contr¢cl Documents (ICD). DHE E PULP OSS Of F415 


process is to provide a timely and complete definition of 
interface details and © provid= a continusus in-depth 


review process [ Ref. 29]. 


(5) ee Lescing. The Quality Assurance role in 
Mae actual testing 1s wsinor. The bulk of the Quality 


Assurance groups work is Gone prior to actual testing. 
Quality Assurance First identifies exactly 
what is to be tested andat the same time develops 32 
detailed definition of the test environment. They then 
explicitly define both the test and the evaluation process, 


especially success/failur2 criteria. 


o2 





During wicmereecc ul | Se Sc ans 
MesWibance ects as a MOnatd=> to ensure that che previcusly 
defined procedures are being followsi. They also = 
any changes are documented correctly [Ref. 29]. 
(6). Problem Revorting and Review. Any prob- 
lens uncovered during the testing orocess are documented 
using a Discrepancy Report (DR). Ince testing is complete a 
post-test meeting is held and the Discrepancy Reports ar?3 
assigned to individuals for resolution. The Software Qualit 
Assurance group monitors all outstanding D&s t> ensure that 
all probiems are corrected. 

The Discrepancy Reports serve aS a measure 
of product quality. The software QA group analyzes the data 
provided by the DR and prepares a statistical report. fn 
{[Ref. 29] Stephen L. Stamn, Manager of Productivity Programs 
meeene G. ©. Space Diviston, points su= that such things as 
the number of DRs, VremeESGUency d2StrEibucion cE DRS by 


type, the mean time of closure (correction), and the DR rate 


it 
ct 
Oo 


eas @ Function of preduct Life can be used by managemen 


identify weak spcts in the software implementation proce 


Nn 
OP) 
e 


Gj we 2Odsan Library. A System similar to that 
at NOSC is used at the G. E. Space Division. 

(8). Quality Assucance To0ls. Automated soft- 
ware cede analysis tools are used as part of the test 
program to uncover code in the final product which has néver 
been executed. If this is the case it is determined if thers 
is a hole in the test progran, 2 possible fiaw in the 
Peeauce Cesign OF just sone superilusus code in the finisned 
Beoauct [{Ref. 29}. 

(9). Lessons Learned. The following eléments 


[Ref. 29] have been found by the G. EB. Space Division to be 


necessary for a successful Software Quality Assurancea 


Program: 
1. The Software Quality Assurances, Plan must have high 
ERGmicGrmv GS betity, tz must d=fine the SQA program at 
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CwlevVetmordcratly sitticient to allow 


=~ ip 
Soest Nave Ghee EProqgect 2nd conpany 
whole~hearted support. 


ee odo Phe ones So CmolaOQnceWame emgincering tech- 
nigues by the Sao ee te Seaget Specieeceliy targeted 
at increasing product quality. 


3. The project organization must distribute the Software 
Quality Assurance Program responsibility, placing SQA 
tasks where the capability really exists. 

Be Tne abilit to neasure the 2rffectiveness of the QA 
program and, if possible, the quality of *he end 
PEGG Uc tc. 

Be, SOttware Quality Assurance personnelmust pe an 
POeeSGral Pdiaes Ole ene PiGo ject team. 


i 


Boe ONCLUS ION 


Four separate Software Quality Assurance organizations 
have been discussed in this chapter. However they are 
Simiiar in several ways. 

First the Software Quality Assurance aqroups at each of 
these organizaticns becom? involved with the project early 
in its life-cycle. Each of the Quality Assurance groups is 
invoived in every phase of the development process. Bach of 


ieeese OLGanizaticns recognizes the fact that an effective 


(Dp 


Ouality Assurance organization allows them to deliver 43 


quality software product which mneets i Cont =ac. Wak. 
requirements. They are also in agreement on =zhe fact 
Software Quality ASsuranc2 saves money in the development 


Bemees= by identitying and correcti1g errors sarly in the 


(D 


process. They also agree that an effective QA progrzran 
provides both management (Corporate and Project) and «he 
S@eteomer Wh Visibility into the development process. The 
view of software aS a product is also a similarity among 
these organizations. Finally all these groups agree that 
Software Quality Assurance groups do not create quality in 2 
Meemect but that it is in fact part of the job of every 


project member. 
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A. STRUCTURE 


Within the Naval Material Command, ¢the Fleet Material 
Support Office (FMSO) is a field activity sponsored py the 
Naval Supply Systems Command. FMSO parforms two major func- 
tions in fleet legistical support. The one discuss2d here 


is that cf principal Navy Central Design Agency (CDA) for 


automated supply, Financial, feetaenance, and icqiStical 
systems, a process which consumés the mnajerity of FMSO 
rescurces. 


A general déscription of th2 organizational elements 
directly related to the central systems development process 
memeaep.cted in figure 4.1 Weenta the Organization the 
Comptroller Department (Coie 91). nd Management Department 
{C re the two departments that can be considered as 


@ 
m ReewemOTiaf® “Sey CUR Gener tmen=s are. production 


t ty 


O 

th Qa 
4) 

LO 

NJ 

Send 


St 


ey 


@eemcecd OF Support the production eff 


= 


cr 
ay 
J 
Qu 
fy 
iF 
MD 
QD 
Fu 
tt 
(D 
Qu 


on 


7 


rh 


Or 
Peete OEGanizgd OnS which are direstly responsible for + 
nd 


+r 
iD 


aedq ~sAveeomMa ed. “De 


development and maintenance of sta 2 
Processing (ADP) systems {Ref. 39]. 

A system is consider2d to be an organized set of ADP 
hardware, environmental/application software, and documented 
procedures desiaqned to automate the basic management and 
operating processes for a customer sites or group of customer 
Sites with common mission responsibilities. "Documented 
procedures" as used abov2 refers to the applicable ADP 
related and non-ADP related procedures established +o 
support the hardware and software aspects of the system 
maet. 391. 


2/8: 
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Figure 4.1 Organizational Structure. 


The rimary role of the Management Department is to 
coordinates with and Sip pose etre str Cr ts or =he CDA 
Seemgct.On Departments. In this Capacity zhe branch «hat is 
@emeemoes= relevance to this paper is that of the Qualit: 
SemetOl Sranch. The CDA Development Process Model, figures 


S 
eee cretiects ali of the basic steps appropriate to ensuring 
that each CDA tasking raceived by FMSO is effectively 
Managed and results in 2 aigh quality product being released 
for use by the customer. The model covers all projects, 
large and small, new developments or maintenance. However, 
it is anticipated that sone of the steps in the model may 
meeebe applicable to all projects. TrSmerore, an €xnleacic 
decision by the appropriate level of management is required 
in order to ¢xclude process steps datermined not applic2bis 
ommee DIC ject. As the model is followed through, note 
specifically the areas that deal with symbols equating to 92 
Qc and 92 QC Optional {Ref. 40]. 
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Be THE QUALITY FEROCESS 


AS a new requirement is received by FMSO a mechanism is 
activated to ensure that the output produced will meet the 
user's expectaticns. This mechanism at FMSO is called the 
Meiadity process, that is, an 2ttitude that extends from the 
individual programmer all the way through the systems devel- 
opment cycle up to top management. By design, it is 4a 
multi-layered approach to achieving quality that begins with 
the System Development Quality Process (SDQP). Falling 
under the SDQP are all th2 separate requirements that will 
eventualiv vroduce a working ADS. At each stag2 of davelono- 
ment, quality standards are imposed upon ail personnel at 
all levels within the chain of comnand begirning with «he 
Beas Do 1it study and raguirement jiefinition, proceeding 
through the functionai design, conputer design, progran 
development, testing, operations and maintenance, and ending 
Meee, the Program Trouble Reports (PTR). Within each of 
these particular evolutions labeled above are subsections 
that must be completed béfore the process avolves further. 

Mmevenead On cop of the SDQP are the Quality Control 
Mechanisms. These Quality Control Mechanisms provide the 
meee ty ~O ensure that a guaiilty and error free product is 
mie ftact produced. The methods utilized +o accomplish «his 
are good project management during the requirement defini- 
tion and functionai design stage, sound data base management 
during functional design and computer design, an effecrive 
verification process during computer design and ovrogram 
deveiopment, Paovpeteveliat=On —prsecscadures during progran 
development and testing, and a satisfactory prototype/op 
review during testing and operations/ maintenance of 42 


system. 
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Smecowmotr = 70Ge che SDQOP ara Quality Contr 
we have the Quality Production Concept S 
structured processing, standard data element usage, uniforn 
standards, methods and procedures, improved programming 
techniques and training. When ali of these various layers 
are utilized and implememted as a whole we have the philo- 


sophy of FMSO towards producing a quality product. 


C. QUALITY ASSURANCE VS. QUALITY CONTROL 


Quality Assurance, as defined by FMSO [Ref. 4OJ, is 2 
iiné management responsibiliicy. NemSUGi wy b2hs SUpervesccs 
mes accountable for enforcing the 2zpplication o 
procedures that have been ieveloped for the primary purpose? 
Seen surang accuracy, thoroughness of method, Simplicity ina 
design, adequacy of testing and clarity of documentaticn of 
ADS deveiopment. TO aid all levels of personnel within zhe 
various line departments in the accomplishment of their own 
specific requirements for quality, ‘guidelines have heer 
developed that include NAVSUP PUBs 505, Seem 508, CDA 
DEVELOPMENT HANDBOOK, CDA MANAGEMENT HANDBOOK, TMSO Internal 
Instructions, and various other docunented and undocumerzei 
departmental proced tres. Everything written and documented 
must be in compliance with these standards. It is the indi- 
vidual person's responsibility to snsure that they are in 
compliance with these standards. 

Ouavetcy Control, as defined by FMSO (Ref. 40], is the 
responsibility of the Management Department and specifically 
tha Quality Control Branch. Mialicy Gent tol procedures are 
those actions that are taken as an ADS is being developed to 
insure that all the required QUALITY ASSURANCE procedures , 
those actions performed by the line devartment personnel, 
toe be cCofmplied iwith to produce a reliable and error free 


RDS product. Quality Control than is primarily a review 
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function to be performed by the Quality Control Branc 
ena s area they are responsible hOp Mensia ing that 
designated/high interest ADS projects conform with sta 
of completeness, accuracy, clarity, and all other applica 
quality standards that applied to the line quality a a 


program have peen achieved. 


D. SPECIFIC QUALITY CONTROL RESPONSIBILITIES 


As quality control is a review function and given thse 
limited rumber cf personnal assign2i to the branch, not 


Seem OUD. 222 is provided by FNSO will be reviewed by 
BS : = aa —>-~ - ? > = ~—- = wie we —_ = —_ i 2 lle te 


em OUuality Control Branch. Paew Olalissy Control Branch 
will, however, be d=zect ily invelved with those orojects 
designated high pricrit GE Gf ) Sdbecial interest to the 
Command. Rie ther projects will bs reviewed on a priority 


Baees af the manhours that can be jievoted to the futher 
enhancement of the quality effort become available. Thess 
area of responsibliity is 2normous and their tasks numerous. 
Therefore, only areas of major responsibilities are listed 
below [Ref. 40], 


1. Review of Functional Descriptions for compliance with 
standards. 


2. Participate ina system design lenis u 
the design has considered all of the proposed 
requirements. 


Be Review of System Specirications for compliance with 
standards. 


YW. Review of Program Specifications. 


5. Review of the Maint2nance Manual, Operations Manual, 
and all other applicable nanuals. 


6. Prepare an.analysis of PTRs_ received as to caus? or 
symptom and recommend possible corrective action. 


7. The Quality Control Branch will _selectively rev 
tes— plans and tests for compliance with Qual: 
Assurance guidelines. In the eee ace Qi ve 
task, ua may desk check all applicable data or they 
na alec to attend the review conference held 
between the programmer and analyst as they discuss 
the results of the test. 


el he 
He t(D 
<= 


8. Review the Implementation Plan. 
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Tee OUdtey Comer Ol 925 LTespons* bis for post implementa- 
t2oOneevwow <S 2O Selected Lies of Tesora scene CTs 
to determine whether the product released is working 
Satisfactorily, meets the nesds of the user, and is 


being utilized correctly. 


10. Perform a Quality Assurance Review of all designated 
ADS programs. 


Es TESTING TO ENSURE QUALITY 


The testing process involves many different personnel 
and includes many different responsibility assignments. For 
example, the individual programmer assigned to the project 
is responsible fcr reasonable testing for all error condi- 
memes chat COUld occUr in the progjrem and for providing 
Support for the systems t2st requir2d for the application. 


S 
Th2 Lead Programmer is r2sponsiblea for developing the *est 
fo Ss 


ju 


plan for system testing an eslrkons 


The Systems Analyst is responsible for assisting ‘the 
Lead Programmer in planning and coordinating the string 


Peaecaung/SyYS-em testing to determine that all «he programs ir 


Bee appiication produce the required output when run in 
Eocal. The Systems Analyst has a orimarv Tesponsibiliity of 
approving and/for selecting the test data used for systems 


testing. 

The Systems Designer is +t0 participate in reviews of 
test output to insure that testing of the ADP program has 
been adaquete. After the processes above are compieted, thea 
Quality Control Branch will selectively revisw test plans 
and test results for compliance with all Quality Assurance 


Guidelines. 


1. Types Of Te EOL Oe| 


Sr EY = = 


During the. testing cycie there are primarily four 
dafferent methods utilized to check the programs. 
1. Unit Test - A single program that is checked by the 


programmer responsible for also writting the code. A 


cae 





Unit Test Review is scheduled between the anaiyst and 
programmer with che *est cesults being freviswee to 
ascettainmrr futher testing is warranted. 

2. String Test - Each program release which requires the 
execution of other programs in actual production will 
Se st=ings tesced. Programm2rs are responsible for 
writing the test plan to ensure that all major paths 
and functions that will utilize the new program are 
checked. 

3. System Test - Each new ADS or major change involving 

n 


more than one application/oneration or package in an 


SS. 00 oboe Nee Oe SUD jJeCt=i Oa System cest that 
wll evaluate the specific system as 2 whoie. The 


OVencweIeresponet bil Sy £or th> test will De giver to 
the Lead Frogrammer. The Quality Control Branch will 
evaluate the results to ensur2= that the system mects 
design objectives. 

4. Integrated Systems Test - Thi 
to test all program interfaces, dat 
and ali internal and eX? 
correctness of data flow. Tha 
Lead CDA Department are responsible for 
formal test plan and conducting the test. The Quality 


Control Branch may or may aot be assigned as 2: 


| 


Sweralbls MONnitOr Et this procedure but will be 


required to review the results. 


FP. SYSTEM RELEASE PROCEDURES 


Upon completion of the required evaluations and for 
specified projects, the Line Departments involved will 
forward to the Quality Control Branch 211 applicable docu- 
mentation for review. rter this review has been completed 


the complete package is sent back to the CDA Departmen= Line 
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Manager. When the Manager is satisfied that the orsoaran 
mests the requirements ani that ail quality assur 
dards have been met, the Line Managarc will, by his sign 
release the program for use. This will then te 


Systems Development Process 


G. EVALUATING THE QUALITY PROGRAM 


im omnportant tO .realuze that at FMSO, QOUality Control 
moenot antimately concerned with the daily operations of <he 
line deépar«ments,. but is concernei with assessing the 
results of the line departments Ween chas perspective in 
Mand, «he design and execution of the Quality Assurance Pl 
fem cons:dered to be an integral ovart of the prod 
process itself. Quality 2Sssurance for software cons 
the formal application 92 standards and execution 
required tests, -and then the assessment of results. 
Enforcement of the standards and ths necessary adjustments 
Memene Drocess is one of the responsibilities of «he Quaiicry 
SemerOl Branch. 

mecnough tie Quality Centrol Bra 


his deeply embedded 
performed in an 


we 

within the Management Department, ‘they ha 
extremely professional manner. When given an adequate 
amount of time and an opportunit SOpperrOsmn In “ehetr 
primary role as reviewers, they have always met «he chal- 
lenge. But this opportunity to excel does not always occur 
as it should, and FMSO is no diffarent than any other soft- 

ware producing organization. As the project completion date 
draws near, usually the first item to be called excessive <9 
mies DLOgGram 1S quality control. Dae oGnas he pro j}ec= 
Manager is respcnsible for ensuring that enough time is 
mpeored during the SDP so that Quality Control has a chance 
to evaluate the program. This estimate of the amount of 


tame at will take to complete a project is a most difficult 
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one, but dates have to be set and, on occasicn, 
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ft 


ct 


hej 
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o = s ~ — ms 7 &e mn oP mH, 4 ~ 
emmeenctS are unable to cite specifics Examples o 


Gemctro! being cut Short, bat when the amourt of work <hat i 


Ui 


to be accomplished and the number of personnel assigned +o 
accomplish it are compared, the assumption can be made that 
=* has occurred , either as a result of internally or exter- 
nally generated pressure £9 comply with a due date. 

When the value of a product is very hign, such as on 


command designated /special interest projects, e¢ach item 


produced is individually inspected and gone over in fine 
detail at all levels, from the projyrammer +hrough all the 
créviews, and finally top nanagement BOWSVGE, wae Less 
important conditions, thea selection and review process of 
beeetdcts 1S not aS critical, and fot very minor projects i+ 
need not be. This does not mean that the product is anv 
less important to the user but merely implies that all 
projects other than the 2xceptiornally Simple ones should 
also receive the same dieqree of scrutiny that special 


projects do. 


Pe tempo soto ciaes One CcuCctates Chat the 


O 


mie Tesailty 


above process is not feasible at this time. Wee lS Were 


sv 


Sm@eti staff in the Quality Control Branch an4 zhe multitude 
cof tasks asSigned, it is physically impossible to meet all 
of their requirements, and priorities must be established. 
With an cngoing review of the number of Computer Specialists 
that can be justified within the Quality Control Branch, i+ 
is essential that the justification be met and billat 
descriptions constructed *o place more peoples, not less,in 
this most important branch if FMS) is to continue with its 
present emphasis upon producing a quality product. 

The authors feel that ec moO Naas y CON -TOlL/OUel i ey 
Assurance program at FMSO is highly competitive with 
Similiar organizations. Their thorough and most agressive 


instructions and standards are excelient, and if the present 
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level of entorcement Psomcotcanwec Wl lt G=>ea tity @€Nnance ths 
Pr y au 


Sauc: Serving che Ple2t. Sue ls = Contr sae, aliny 
Assurance dces happen at FMSO, Peay due to. ene Sttuc- 


tured process and to the professionals that work and manage 
jhe Organization. The degree to which it happens is an 
evolving entity. 

At the present time th2 only affective documented method 
to measure the application of quality practices within FMSO 
is the analysis. and evaluation of the Program Troubles 
Reports (PTR). FIRS may b=? submitted during any step of thea 
systems development process or after the program has been 


Seal 
nr ee 8 


MecoG one ~1elq user. Weesa S2 ee = soca: yan ar rMSO 2+ 


routed to the Project Control Branch where is is logged in 
re) 


O 18 
and then sent the department that issued the program with 
Waecn the PTR 23s concernei. The department inveived then 
@eecidges =f the PTR is asf a critical or non-critical nature 
and then proceeds *o work on 1%. Pu 


two ways, critical, which has a sig impact upon 


xz 3 WN A 

{-3- 
ih 
1" 
Q 
pel 

D) 

t 


daily routine, ange non-critical, which has less of an impact 
ed. Critical PIrRs will be 


and can temporarily be delay 
corrected as socn as possible an 


normally within <«hree 


th 


ol 
WGEKing days from receipt of sufficis 
W 


Me anrormac: On eeomel low 
=o CDA to act. Non-critical PTRs 1 


1 be corrected as soon 
as possible after receipt of surficient data that will allow 
the CDA to act. 

The Quality Control Branch performs a quarterly analysis 
of all PTRS received with special emphasis on “he mos+ 
common type of problem, which st¢ps of the systems develop- 
fiem@re DLOCeSS Creates the most errdsrs, identification of 
trends, and if possible, recommendations for corrective 
actions. 

A sutvey of -the PTR Analysis Report for the Second 


Quarter CY 82 reveals the following: [Ref. 41], 
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1, 


PTO catemGt eS S65 020RS, Of Watch 275 were c 
WeromesmGe! Locreasa 1a treciassi fied. Th 
were due to either being invaialid or airsady in exis- 
tence. A reclassified PTR was one that, when 
Submitted, it was felt by the user that the program 
was not performing as desired or requested but upon 
researching the problem it was found that all 
requirements had been mex. To meet the new require- 
ment of the user 2 new project would have to be 
desiaqnated. 

Of the coffPleted PTRs, 38 were criticai and 237 non- 
Grew Ga). 

The FMSO average number of manhours +o fix a PTR was 
VemeboOn Oo sche ocat PTR and 21 £0T a Non-=critical PTR. 
One possibl2 reason for the difference in times is 
the experience level of personnel assigned to repair 
a program. A more experienced programmer is géener- 
ally assiaqned to a critical PIR. 

The average number of days for FfMSO to comvlets a 
Sara Galt etR. 2S 15.5 S28de51.6 ~€O0 ccmplete a non- 
C=aiecale Ft. This figure may be skewed toward <=he 
high side very easily because of the small number of 
Gritical FTRs but doses bear close monitoring. 

OZ the completed PTIRs, 57 percent were caused by 
coding/design errors. 21 vercent were classified as 
other and not designatei to any category. However, 
42 percent of the criticai PTRSs were caused by 
PEOdpam OT COding Srrors. St2ps have been taken to 
better divide the cause category in an attempt to 
better evaluate the ¢rrors that were classified in 
the other category. 

The comparative completion tates for critical PTRs 
has remained relatively stable compared to the past 


year. Hewever, the non-critical completion rate has 
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miemcaced sSubScancriadiy and ths trend 15 fcr arn 
greater completion rate. 

The Received-rResoived-dutstaniing results sh 
POURS 
PTRS resolved 
218 Palojetel 


outstanding PTRS is now in 2 


the number. of Reece vod ts 


number of is increasing 
of 


steady decli: 


and asa result these 


The number. of programs released increased about 12 


percent over the pr2vious quarter and yet the number 
of PTRs by 
Al+hough the over 


oscillated, 


about 7 


the last 


received decreased percent. 


numoer cf PTRS y2ar has 


the trend shows there to b2 no signifi- 


cant increase and thus a net G@ecrscass af - Saito of 


programs to PTRs should be anticipaed. 


Considering that only nine personnel are assigned to the 


Brarch and that FMSO now has 


@aality Control 


programs in existences, the 


piah appears to 
meporce clearly 
wil ty Control 
be displayed in 
tBe product 


set and an 


programs for a more detailed 


in accomplishing 


report shows an 


effective method of measuring 


ro; 000s pais 
Quality Control/Quality Assurance 
ke headed in the right 


@zrec*20n es the P 


ah 
continued  ampnasis on the 


shows. AZEC Ss 
effort an even more effective program will 
ThCmer UT Utes 6 To achieve a 100 percent error 


the ideal but 
and selecting the 
Dives da vtorewell @azdq greasly 
the qoaiL FMS) The PTR 


improvement over preceeding year but 1% 


sets for itself. 


the 


also quite effectively shows other areas that may need addi- 
tional emphasis. In the following chapter we will list some 


@Seecune areas that we feel should b2 tsOonsidered in the future 


corporate growth pattern of FMSO. 
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Ve RSCOMBENDATIONS 
This paper has been a2 compreh2nsive investigation of 
software quality assurance from a theoretical viewpoint and 
from specific investigation into the quality assurance/ 
quality control departments of varioas organizations. Based 
on the authors! research, the following recommendations are 
offered in hopes of enhancing the Fleet Material Support 

teerce*s (FMSO) quality control effort. 


Vi 


ammreceos Ore == base 2224 concerning *he cuality proces 
needs to te established. Before a specific direction 
Can be maintained, a mechanism must exist that would 


enable FMSO +o m2asure its ability > meet the 


ct 


jy) 


desired objectives. hoes felt “that; <a mininun, 
m 


the Quality COn-c Ole etancn could 2xamine oast 


programs and select at least two that were very 
Similar fer an analysis as to the effectiveness of 
woe Gialeasty COREEFO. program. Th2 crite2ria of these 


*wo programs Might be (1) that they were produced by 


the same department within a short ‘time period of 
each other, (2) that they were intended to be 
utilized in the same fashion, and (3) Chateweoney or 


=hem had been reviewed by Quality Control throughout 
the entire process and the other had not received 
this same critical review. 

2. Top management must continuS +o re-emphasize the 
Biporeames Of Guatity control within the organization 


and display a positive philosophy of commitment to 


the quality process. As has been discussed, without 
top management support, quality assurance/quality 
control programs produce a less than desirable 
CUED lc. 
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The number of parsonnel assigned to tne quality 
COmaea ss teamien 15 deiin ly 
organization as large as FMSO, which produces 800 
plus programs per yuarter, a quality control organi- 
zation of only eight people plus a supervisor can not 
reasonably be expected to meet their obligations and 
responsibilities on all occasions. The Quality 
Control Branch needs an infusion of personnel. 
Vee on tome nOUldsbe paid not onky to the quantity of 
personnel but also to quality. As stated previously, 
the core of these personnel need to be as knowledge- 
able as senior systams analys*s and also command the 
respect of the individuals whose systems are being 
evaluated. 

A method of augmenting the staff of the Quality 
Con. 50 L Tanch £rom an external sou 

Temiige ste edSe adn indocctsination <£ z 
Puteccoemeen COre Of highly gquaiified personnel could bs 
Maintained .that were the permanent part cf the 
Cucieey Ccarecl Branch. New hires could be given 
Boas tetmonasy DOSUEIOn and casked with such voroj 

Sse rcViewend ehe PErOgect SPescicications, all of tne 
manuals, and all other applicable materiai. This 
would alsc afford the new personnel an opvortunity to 
receive formal and correct training before being 
assigned eRe, the ba rtecular line department. 
Additionally, it would enable the new hires to gain a 
better overall understanding of what the organization 
dees and the amount of interfacing that must 
conducted before a project is completed. Of courses 
there would have to be some discrationary measures 
imposed when selecting personnel and atime linit 


must be adhered to. 
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Gemctdesat2on Showla be given to moving the Quality 
Centrol Branch out crits pr2sent management s=ruc- 
Mhe authors feel that, because of the massive 
size of the organization and enormous amount of 
Material that must be reviewed, the Quality Control 
Branch should be moved so as ¢9 have at least parity 
with line management. This position may be desig- 
Rated as Quality Control Department Head, Code 99. 
Billet descriptions weuld have to be re-written and 
the management problems overcome, but the increased 
4 


Comminicaeons, be Weney voluntary or “by direction, 


petween quaiity control and iine management woul 
have an impact upon future products. Additsepaliy, 
—hasS WOW dr at ford =< he qualic Cont rEed ersonneél an 
easier avenue to b2come nore directly involved with 
their responsibilit of ensuring that standards are 
met and better ?23nable tham to enforce theses 
Standards. 

Line managers must be educated as to the importance 
Ome DO MGMlal Lt y@eont hel ITunction. zn this vein, when 
@penangS abise in the Quedity Ccntrol Branch, quals- 
fied line personnel should be encouraged te apply for 
the positions. The benefit gained for the organiza- 
tion would more than offset the loss =o one 
individual department. ho ssuppOrTe » the attitud: ral 
change that must take placs, an ongaangq thaining 
preogram, sponsored by top management, will have to be 
*mplemented on a company wid2 basis and the positive 
benefits gained from the addition of these qualified 
Pesseonnel =o the Quality Control Branch must be 
discussed and dispia yed. 

Quality control checklists nust be ré-instituted. 
There should be a ganeral checklist applicable to all 


programming functions as well as specific checkiists 
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10. 


Seqacdaimeg —edeh oni. Vrdual program. fewopeuld be 
emphasized that thase checklists are +9 be used as 
BCeeGem= “OL aoeas  COLSccruee2y> tocieto reint 
standards, rather than a method of laying blame. 
Lee enOOnc ant ©Or scOnt  nulty shat there be consis- 
tent assignment of quality control personnel to 
Puowects. One person (or team of persons) should bs 
assigned to a project, except the smallest ones, and 
should monitor this project all the way from incep- 
tion to post implementation review. 

A better method, possibly 2 iecision support svsten, 
must be devised to determine which projects wili 
Uedeadomaidiray sont col! PrOscsiursas. fe is recognized 
hat the number of projects may preclude all projects 
being scrutinized by quality rtontrol. However, «here 
needs to be a better system to ssiect programs for 
review and evaluation than the present method of 
committee or command discretion. Curren ly the 
Paetabywete= nod fOr evalueep~ag projees length and 
Genpl=irty 1S experience. TRS "SUbAeCCII VS Viemnolae 


is the basis for the depth of involvement achieved by 


Gialtaety @ontrol. A logical and comprehensive deci- 
S2en precess will serve to)060alleviate CEisas 
Management. and ensure that the mos= important 


projects are chosen. 
End users must become even nore involved in the 


design and development of software throughout the 


igftecycle. Users shouid work closely with quality 
control +o ensure a timely and correct review 
process. There must also be better communications 


established between users and line programming func- 
tions. This communication may ke facilitated bv 
quality control andi may help lower project trouble 


report (PTR) reclassifications. 
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Relirs 


= * 


Ce 


O 


+ 


licax andmeoncis= documentation 2s an eng 


OM 


O 
Peer SCEoWanbe p=OdUGCT2On Eacility. The quality 
cortrol division muect review documentation, ore 
coding, and assis where possible to impreve any 


deficiencies. 


12. Currently, the program troubles reports (PIR), are the 


Wl oue 


primary documented measure of the effectiveness of 
the quality program. Whils this may be a valid 
measurement, there needs to be a search for addi- 
tional measures. SiGe sduality conerol does not 
impact 100 percent of the programs leaving FMSO, 4 


“= 


mezthod or evaluating the e say (6 2 (vo(gbl: A Usliea Maeno) pte a) emo el 
projects which do not have quality control involve- 
ment must be formulated. 

The principle of top-down design and tcp-down testing 
should be reevaluated. AS an altermative, top-down 
design and hettom up testing should be considered. 
7 


1 feel] une 


mw 
tn 
Ww 
v7) 


u 
Tt is assumed *hat top nanagenen* wi 
fe) 


the present process is shangei. 4H 


Shown ila many studias that errors ma Ee eaee Geass 
phase of a project are the most expensive to repair 
because one has to return back to the beginning and 
ie ceraliy begin tha entire process cver. 
Requirements must be reevaluated, specifications 
redone, coding rewritten, and finally, the progran 
must be tested again. Since the majerity of design 


Exrors are not found until the testing phase, it is 
easy to see that the amount of extra time spent in 
che design phase will, over time, offset any anxiety 
that top management would have because of the seen- 
ingiy lack of progress on the project. 15 cea = 
Suggested that bottom-up tasting will force the 


design effort to inprove. 


es 





14. 


TS 


Phased deveiopment should be adhered to. Coding nust 
not begin until after the design is comp 

Ge=en¢e se the lead proeqssnn n 

pregrammers to the difficult nodules and an inex 
enced programmer to the $3 r modules/prog S 
fois eeere Or  cCholgnt should fiiter throughout the 
project and allow for better personnel utilization 
but should also fford the parsons writing the test 
plan to do so in a nore reliable manner. In allowing 
the writing or code before the final design is 
completed, -.we achisve the short term goal of being 
aclie te sé@ a working product that can be tracked on 
Smee ale. However, we cannot ses the long range goal 
of whether the module will produce the required 
output or the required number of interfaces. 

The system to trace the phase of development for each 
preject should be reemphasized and qualit SOnmeso | 
MuS= continue to o2 kept abreast of alli the current 


production efforts 


jes 
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